Jul 14 18:19:28.085145 ip-10-0-131-199 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 14 18:19:28.085158 ip-10-0-131-199 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 14 18:19:28.085167 ip-10-0-131-199 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 14 18:19:28.085503 ip-10-0-131-199 systemd[1]: Failed to start Kubernetes Kubelet. Jul 14 18:19:38.254650 ip-10-0-131-199 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 14 18:19:38.254673 ip-10-0-131-199 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6e796b496414415e9d492c88a1463fab -- Jul 14 18:21:48.153726 ip-10-0-131-199 systemd[1]: Starting Kubernetes Kubelet... Jul 14 18:21:48.477087 ip-10-0-131-199 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:48.477087 ip-10-0-131-199 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 18:21:48.477087 ip-10-0-131-199 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:48.477087 ip-10-0-131-199 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 18:21:48.477087 ip-10-0-131-199 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:48.478858 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.478781 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 18:21:48.485966 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485945 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:48.485966 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485963 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:48.485966 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485967 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:48.485966 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485971 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485974 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485977 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485980 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485982 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485985 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485988 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485990 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485993 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485995 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.485998 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486001 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486003 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486005 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486008 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486010 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486013 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486015 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486018 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:48.486125 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486021 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486023 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486026 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486028 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486031 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486033 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486038 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486042 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486045 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486048 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486050 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486053 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486055 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486058 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486062 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486065 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486068 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486070 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486073 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:48.486602 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486076 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486078 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486081 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486084 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486086 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486089 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486091 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486094 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486096 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486099 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486102 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486105 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486107 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486112 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486115 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486118 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486121 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486124 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486127 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486129 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:48.487086 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486132 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486135 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486137 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486140 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486142 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486145 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486147 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486150 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486153 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486156 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486159 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486161 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486164 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486166 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486169 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486173 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486175 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486177 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486180 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486183 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:48.487582 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486185 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486188 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486191 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486194 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486196 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486198 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486201 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486203 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486205 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486208 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486210 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486214 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486218 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486220 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486223 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486226 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486228 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486231 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486949 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486956 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:48.488057 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486962 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486965 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486969 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486973 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486976 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486979 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486982 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486985 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486989 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486993 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486995 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.486998 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487001 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487003 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487006 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487009 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487011 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487014 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487016 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:48.488547 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487019 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487021 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487024 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487026 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487029 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487031 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487034 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487037 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487039 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487042 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487044 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487048 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487050 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487055 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487057 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487061 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487063 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487065 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487068 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487070 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:48.489115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487073 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487076 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487078 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487081 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487083 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487086 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487088 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487091 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487093 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487096 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487099 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487102 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487104 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487107 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487109 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487111 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487114 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487116 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487118 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:48.489626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487122 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487125 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487128 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487132 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487134 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487137 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487140 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487142 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487145 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487147 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487150 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487152 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487154 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487157 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487159 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487162 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487164 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487166 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487169 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487171 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:48.490112 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487173 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487176 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487178 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487181 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487184 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487186 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487188 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487191 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487195 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487197 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487199 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487202 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487204 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487207 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487209 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487212 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487215 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487217 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487219 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487610 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:48.490622 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487615 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487618 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487620 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487623 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487626 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487628 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487631 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487633 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487636 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487638 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487641 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487643 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487646 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487649 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487651 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487660 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487662 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487665 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487667 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487670 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:48.491117 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487672 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487675 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487677 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487680 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487682 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487685 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487687 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487690 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487692 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487695 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487697 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487700 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487702 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487705 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487708 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487710 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487713 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487715 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487717 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:48.491610 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487720 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487723 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487727 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487729 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487732 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487734 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487737 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487739 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487742 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487746 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487748 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487751 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487753 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487755 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487758 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487760 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487763 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487765 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487768 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:48.492079 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487770 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487773 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487777 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487781 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487784 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487786 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487790 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487792 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487795 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487798 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487801 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487803 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487806 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487808 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487811 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487813 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487815 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487819 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487822 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:48.492549 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487824 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487827 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487829 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487831 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487839 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487841 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487843 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487846 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487848 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487851 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487853 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487855 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487859 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487864 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487867 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487870 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487873 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487875 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487878 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487881 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:48.493001 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.487884 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.487960 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.487970 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.487979 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.487987 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.487993 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.487997 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488003 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488009 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488012 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488015 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488018 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488021 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488024 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488027 2580 flags.go:64] FLAG: --cgroup-root="" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488030 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488033 2580 flags.go:64] FLAG: --client-ca-file="" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488036 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488039 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488043 2580 flags.go:64] FLAG: --cluster-domain="" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488046 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488048 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488051 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488054 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 18:21:48.493493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488058 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488063 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488065 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488068 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488071 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488074 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488077 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488080 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488083 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488087 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488090 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488093 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488096 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488099 2580 flags.go:64] FLAG: --enable-server="true" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488101 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488106 2580 flags.go:64] FLAG: --event-burst="100" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488109 2580 flags.go:64] FLAG: --event-qps="50" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488112 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488115 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488117 2580 flags.go:64] FLAG: --eviction-hard="" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488121 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488124 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488127 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488129 2580 flags.go:64] FLAG: --eviction-soft="" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488132 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 18:21:48.494057 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488135 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488138 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488141 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488143 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488146 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488149 2580 flags.go:64] FLAG: --feature-gates="" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488152 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488155 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488160 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488163 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488166 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488169 2580 flags.go:64] FLAG: --help="false" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488172 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488176 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488178 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488181 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488184 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488188 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488191 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488193 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488196 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488199 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488201 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488204 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 18:21:48.495133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488207 2580 flags.go:64] FLAG: --kube-reserved="" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488209 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488212 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488215 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488218 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488220 2580 flags.go:64] FLAG: --lock-file="" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488223 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488226 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488229 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488234 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488238 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488241 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488243 2580 flags.go:64] FLAG: --logging-format="text" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488246 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488249 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488252 2580 flags.go:64] FLAG: --manifest-url="" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488255 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488260 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488263 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488267 2580 flags.go:64] FLAG: --max-pods="110" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488269 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488274 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488277 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488280 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488283 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 18:21:48.496177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488286 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488289 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488295 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488298 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488301 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488304 2580 flags.go:64] FLAG: --pod-cidr="" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488307 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488311 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488314 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488317 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488320 2580 flags.go:64] FLAG: --port="10250" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488322 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488325 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d2b963651e399480" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488328 2580 flags.go:64] FLAG: --qos-reserved="" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488331 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488334 2580 flags.go:64] FLAG: --register-node="true" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488336 2580 flags.go:64] FLAG: --register-with-taints="" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488340 2580 flags.go:64] FLAG: --registry-burst="10" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488343 2580 flags.go:64] FLAG: --registry-qps="5" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488346 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488349 2580 flags.go:64] FLAG: --reserved-memory="" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488352 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488355 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488358 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488361 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 18:21:48.497258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488364 2580 flags.go:64] FLAG: --runonce="false" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488367 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488370 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488374 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488377 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488380 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488382 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488385 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488388 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488391 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488394 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488397 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488399 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488402 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488404 2580 flags.go:64] FLAG: --system-cgroups="" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488407 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488412 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488415 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488417 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488421 2580 flags.go:64] FLAG: --tls-min-version="" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488423 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488426 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488429 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488431 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488434 2580 flags.go:64] FLAG: --v="2" Jul 14 18:21:48.498337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488438 2580 flags.go:64] FLAG: --version="false" Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488442 2580 flags.go:64] FLAG: --vmodule="" Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488445 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.488448 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488578 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488582 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488585 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488588 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488591 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488593 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488597 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488599 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488602 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488605 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488607 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488610 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488612 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488615 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488617 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488619 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:48.499214 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488622 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488624 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488627 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488629 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488631 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488634 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488636 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488640 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488643 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488646 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488649 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488652 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488654 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488657 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488660 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488663 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488666 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488668 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488672 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:48.500096 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488675 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488677 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488679 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488684 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488686 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488689 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488691 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488694 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488698 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488701 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488704 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488706 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488709 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488712 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488715 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488717 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488720 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488722 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488724 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488727 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:48.500654 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488730 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488732 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488734 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488737 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488739 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488742 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488744 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488746 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488749 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488751 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488754 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488758 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488760 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488763 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488765 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488769 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488771 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488774 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488777 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488779 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:48.501282 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488781 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488784 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488786 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488789 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488791 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488794 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488796 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488799 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488801 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488803 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488806 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488808 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488811 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488814 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488818 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488820 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488823 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488825 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488827 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:48.501990 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488830 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488832 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488835 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488838 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.488842 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.489298 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.496723 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.496743 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.496767 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.496782 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496886 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496893 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496899 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496903 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496907 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:48.502556 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496912 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496916 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496920 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496925 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496929 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496934 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496939 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496945 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496951 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496956 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496961 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496966 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496970 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496974 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496979 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496983 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496987 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496992 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.496996 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:48.503006 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497008 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497013 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497016 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497021 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497025 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497030 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497034 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497038 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497042 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497047 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497051 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497055 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497059 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497063 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497067 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497071 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497076 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497079 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497083 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497087 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:48.503947 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497091 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497095 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497099 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497103 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497108 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497111 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497123 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497127 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497131 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497135 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497139 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497143 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497147 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497158 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497162 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497166 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497170 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497175 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497179 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497183 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:48.504817 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497188 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497192 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497196 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497200 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497204 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497208 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497212 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497216 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497220 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497224 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497228 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497232 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497236 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497240 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497246 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497252 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497256 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497260 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497264 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:48.505698 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497268 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497272 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497276 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497280 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497284 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497288 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497292 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497304 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497308 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497312 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497318 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497324 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497328 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497333 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497336 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:48.506403 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497340 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.497349 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497573 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497582 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497586 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497593 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497599 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497604 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497608 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497614 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497619 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497624 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497629 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497633 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:48.506890 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497637 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497641 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497646 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497650 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497654 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497658 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497662 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497666 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497670 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497675 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497679 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497691 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497696 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497702 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497706 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497710 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497715 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497718 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497722 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497727 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:48.507300 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497730 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497734 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497738 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497742 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497747 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497751 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497755 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497759 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497763 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497767 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497771 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497775 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497779 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497783 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497787 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497792 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497796 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497800 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497804 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497808 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:48.508032 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497812 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497816 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497820 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497824 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497836 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497841 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497845 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497849 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497853 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497857 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497861 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497865 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497869 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497873 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497877 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497881 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497885 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497889 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497893 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497897 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:48.508592 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497901 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497905 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497909 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497913 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497918 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497922 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497927 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497931 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497936 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497940 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497944 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497948 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497952 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497956 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497960 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497965 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497969 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497983 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497987 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:48.509070 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497991 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497995 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.497999 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.498004 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.498008 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.498012 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.498016 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:48.498020 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.498028 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.498182 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.500772 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.502093 2580 server.go:1019] "Starting client certificate rotation" Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.502214 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:21:48.509577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.502249 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:21:48.509900 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.504694 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:21:48.513474 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.513441 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:21:48.524776 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.524757 2580 log.go:25] "Validated CRI v1 runtime API" Jul 14 18:21:48.526847 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.526832 2580 log.go:25] "Validated CRI v1 image API" Jul 14 18:21:48.527671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.527655 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 18:21:48.529830 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.529808 2580 fs.go:135] Filesystem UUIDs: map[0beb4680-821f-4125-b7af-7d3da68f40a5:/dev/nvme0n1p4 5bc66924-0ed2-460c-9099-ae5d8bf7ce1b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 14 18:21:48.529899 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.529831 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 18:21:48.533487 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.533448 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:21:48.534761 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.534653 2580 manager.go:216] Machine: {Timestamp:2026-07-14 18:21:48.533528983 +0000 UTC m=+0.281968805 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3111308 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec233f65d1043dd74ea694ff52e9eb0b SystemUUID:ec233f65-d104-3dd7-4ea6-94ff52e9eb0b BootID:6e796b49-6414-415e-9d49-2c88a1463fab Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:8b:c5:b3:4e:2b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:8b:c5:b3:4e:2b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:19:bc:68:1d:97 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 18:21:48.534761 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.534757 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 18:21:48.534843 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.534829 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 18:21:48.537502 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.537477 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 18:21:48.537628 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.537505 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-199.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 18:21:48.538578 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.538568 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 18:21:48.538612 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.538580 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 18:21:48.538612 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.538588 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:21:48.539089 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.539079 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:21:48.540588 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.540579 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 18:21:48.543183 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543173 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:21:48.543301 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543292 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 18:21:48.543330 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543321 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 14 18:21:48.543362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543332 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 18:21:48.543362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543345 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 18:21:48.543362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543355 2580 kubelet.go:398] "Adding apiserver pod source" Jul 14 18:21:48.543439 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543372 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 18:21:48.543769 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543756 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:21:48.543816 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.543778 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:21:48.545545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.545530 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 18:21:48.547327 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547309 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 18:21:48.547374 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547345 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 18:21:48.547518 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547508 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 18:21:48.547568 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547527 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 18:21:48.547568 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547536 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 18:21:48.547568 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547545 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 18:21:48.547568 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547554 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 18:21:48.547568 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547563 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 18:21:48.547944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547935 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 18:21:48.547974 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547945 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 18:21:48.547974 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547951 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 18:21:48.547974 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547957 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 18:21:48.547974 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547969 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 18:21:48.548076 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.547976 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 18:21:48.548581 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.548572 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 18:21:48.548581 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.548581 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 18:21:48.551839 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.551815 2580 server.go:1268] "Started kubelet" Jul 14 18:21:48.552008 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.551947 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 18:21:48.552069 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.552051 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 18:21:48.552117 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.552003 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 18:21:48.552913 ip-10-0-131-199 systemd[1]: Started Kubernetes Kubelet. Jul 14 18:21:48.555235 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.555193 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 18:21:48.558587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.558569 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 14 18:21:48.558948 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.558924 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 18:21:48.559384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.559368 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-199.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 18:21:48.559429 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.559384 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 18:21:48.559597 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.559574 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 18:21:48.560118 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.559349 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-199.ec2.internal.18c23b082496bc2e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-199.ec2.internal,UID:ip-10-0-131-199.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-199.ec2.internal,},FirstTimestamp:2026-07-14 18:21:48.55177323 +0000 UTC m=+0.300213052,LastTimestamp:2026-07-14 18:21:48.55177323 +0000 UTC m=+0.300213052,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-199.ec2.internal,}" Jul 14 18:21:48.562551 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.562534 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 18:21:48.562646 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.562550 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 18:21:48.562716 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.562699 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:48.563157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563137 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 18:21:48.563157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563140 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 18:21:48.563297 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563167 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 18:21:48.563297 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563287 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 14 18:21:48.563381 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563300 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 14 18:21:48.563753 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563739 2580 factory.go:153] Registering CRI-O factory Jul 14 18:21:48.563753 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563754 2580 factory.go:223] Registration of the crio container factory successfully Jul 14 18:21:48.563874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563799 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 18:21:48.563874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563808 2580 factory.go:55] Registering systemd factory Jul 14 18:21:48.563874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563814 2580 factory.go:223] Registration of the systemd container factory successfully Jul 14 18:21:48.563874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563830 2580 factory.go:103] Registering Raw factory Jul 14 18:21:48.563874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.563844 2580 manager.go:1196] Started watching for new ooms in manager Jul 14 18:21:48.564328 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.564316 2580 manager.go:349] Starting recovery of all containers Jul 14 18:21:48.565357 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.565338 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjlgj" Jul 14 18:21:48.572967 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.572824 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tjlgj" Jul 14 18:21:48.574119 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.574102 2580 manager.go:354] Recovery completed Jul 14 18:21:48.575393 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.575366 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 18:21:48.575500 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.575403 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-199.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 18:21:48.578868 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.578855 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.582276 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582260 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.582337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582289 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.582337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582299 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.582874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582848 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 18:21:48.582874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582863 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 18:21:48.583065 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582882 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:21:48.583065 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.582900 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.583586 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.583568 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.583648 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.583602 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.583648 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.583615 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.583935 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.583923 2580 policy_none.go:49] "None policy: Start" Jul 14 18:21:48.583991 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.583942 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 18:21:48.583991 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.583956 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 18:21:48.584856 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.584844 2580 policy_none.go:47] "Start" Jul 14 18:21:48.620001 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.619987 2580 manager.go:337] "Starting Device Plugin manager" Jul 14 18:21:48.620060 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.620014 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 18:21:48.620060 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.620023 2580 server.go:87] "Starting device plugin registration server" Jul 14 18:21:48.620210 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.620197 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 18:21:48.620279 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.620213 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 18:21:48.620330 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.620309 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 18:21:48.620411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.620393 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 18:21:48.620411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.620410 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 18:21:48.620856 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.620836 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 18:21:48.620922 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.620890 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:48.720868 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.720847 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.721917 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.721900 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.721997 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.721935 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.721997 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.721950 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.721997 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.721985 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.731536 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.731496 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.731536 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.731516 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-199.ec2.internal\": node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:48.752552 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.752519 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 18:21:48.753778 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.753763 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 18:21:48.753870 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.753799 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 18:21:48.753870 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.753802 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:48.753870 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.753821 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 18:21:48.753870 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.753859 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 18:21:48.756098 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.756079 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:48.854392 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.854372 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal"] Jul 14 18:21:48.854503 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.854378 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:48.854503 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.854449 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.855766 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.855751 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.855844 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.855783 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.855844 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.855798 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.856926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.856911 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.857042 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.857125 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857057 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.857600 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857586 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.857664 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857615 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.857664 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857624 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.857664 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857587 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.857752 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857688 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.857752 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.857702 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.858763 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.858747 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.858809 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.858774 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:48.859407 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.859382 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:48.859504 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.859416 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:48.859504 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.859431 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:48.864447 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.864429 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaac1d1135f2c4810bad5bfddca8e38b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-199.ec2.internal\" (UID: \"eaac1d1135f2c4810bad5bfddca8e38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.864525 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.864455 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfc6a64e8b5609e46d7d0fea51c9ccea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal\" (UID: \"bfc6a64e8b5609e46d7d0fea51c9ccea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.864525 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.864489 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfc6a64e8b5609e46d7d0fea51c9ccea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal\" (UID: \"bfc6a64e8b5609e46d7d0fea51c9ccea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.885036 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.885015 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-199.ec2.internal\" not found" node="ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.889473 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.889445 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-199.ec2.internal\" not found" node="ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.955251 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:48.955230 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:48.965534 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.965515 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfc6a64e8b5609e46d7d0fea51c9ccea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal\" (UID: \"bfc6a64e8b5609e46d7d0fea51c9ccea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.965594 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.965538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfc6a64e8b5609e46d7d0fea51c9ccea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal\" (UID: \"bfc6a64e8b5609e46d7d0fea51c9ccea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.965627 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.965591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bfc6a64e8b5609e46d7d0fea51c9ccea-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal\" (UID: \"bfc6a64e8b5609e46d7d0fea51c9ccea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.965627 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.965603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaac1d1135f2c4810bad5bfddca8e38b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-199.ec2.internal\" (UID: \"eaac1d1135f2c4810bad5bfddca8e38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.965627 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.965607 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/bfc6a64e8b5609e46d7d0fea51c9ccea-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal\" (UID: \"bfc6a64e8b5609e46d7d0fea51c9ccea\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:48.965716 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:48.965647 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eaac1d1135f2c4810bad5bfddca8e38b-config\") pod \"kube-apiserver-proxy-ip-10-0-131-199.ec2.internal\" (UID: \"eaac1d1135f2c4810bad5bfddca8e38b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" Jul 14 18:21:49.056366 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.056295 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.156830 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.156798 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.187344 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.187323 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:49.192414 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.192397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" Jul 14 18:21:49.257916 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.257890 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.358449 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.358399 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.421218 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.421195 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:49.459374 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.459349 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.501828 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.501811 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 18:21:49.502329 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.501934 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:21:49.502329 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.501969 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:21:49.560261 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.560238 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.576359 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.576331 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:16:49 +0000 UTC" deadline="2028-02-25 03:09:14.647938798 +0000 UTC" Jul 14 18:21:49.576359 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.576355 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14168h47m25.071586557s" Jul 14 18:21:49.661331 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.661285 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.741498 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:49.741449 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaac1d1135f2c4810bad5bfddca8e38b.slice/crio-47edfd9a47325ce92abbe82388a3b3d1209e42e3d0bf072b56895a5b3c72fa05 WatchSource:0}: Error finding container 47edfd9a47325ce92abbe82388a3b3d1209e42e3d0bf072b56895a5b3c72fa05: Status 404 returned error can't find the container with id 47edfd9a47325ce92abbe82388a3b3d1209e42e3d0bf072b56895a5b3c72fa05 Jul 14 18:21:49.746420 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.746404 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:21:49.757528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.757488 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" event={"ID":"eaac1d1135f2c4810bad5bfddca8e38b","Type":"ContainerStarted","Data":"47edfd9a47325ce92abbe82388a3b3d1209e42e3d0bf072b56895a5b3c72fa05"} Jul 14 18:21:49.761626 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:49.761608 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-199.ec2.internal\" not found" Jul 14 18:21:49.817973 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:49.817951 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfc6a64e8b5609e46d7d0fea51c9ccea.slice/crio-9af0a1103f2b335f950a9bfe3715d2accbb1ea9818057304af5b51f231f38ab3 WatchSource:0}: Error finding container 9af0a1103f2b335f950a9bfe3715d2accbb1ea9818057304af5b51f231f38ab3: Status 404 returned error can't find the container with id 9af0a1103f2b335f950a9bfe3715d2accbb1ea9818057304af5b51f231f38ab3 Jul 14 18:21:49.838425 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:49.838405 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:50.886173 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:50.886151 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" Jul 14 18:21:50.896961 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:50.896935 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:50.905481 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:50.905462 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:21:50.906384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:50.906371 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" Jul 14 18:21:50.927270 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:50.927252 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:21:51.568624 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.568349 2580 apiserver.go:52] "Watching apiserver" Jul 14 18:21:51.577310 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.577284 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 18:21:51.578964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.578926 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-lxtcf","openshift-network-operator/iptables-alerter-9j4j5","kube-system/konnectivity-agent-lzkvk","kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx","openshift-image-registry/node-ca-v9k87","openshift-multus/network-metrics-daemon-5wktq","openshift-network-diagnostics/network-check-target-wc7l6","openshift-ovn-kubernetes/ovnkube-node-9hs2p","kube-system/global-pull-secret-syncer-65qbb","openshift-cluster-node-tuning-operator/tuned-xhwnr","openshift-dns/node-resolver-6fzdg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal","openshift-multus/multus-additional-cni-plugins-k8lfb"] Jul 14 18:21:51.581821 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.581800 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.581928 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.581873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:21:51.583567 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.583543 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.585663 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.585641 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.586297 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.586282 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gmjgd\"" Jul 14 18:21:51.586436 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.586420 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.586504 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.586444 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 18:21:51.587028 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.587009 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.587322 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.587303 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.589470 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.589448 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.589840 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.589817 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 18:21:51.590217 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.590201 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 18:21:51.590300 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.590225 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 18:21:51.590500 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.590484 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.590554 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.590540 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m22vx\"" Jul 14 18:21:51.591216 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.590836 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.591216 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.590978 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-66clj\"" Jul 14 18:21:51.591216 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.591070 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 18:21:51.591407 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.591293 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 18:21:51.591534 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.591510 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.592328 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.592308 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 18:21:51.592433 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.592317 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.594410 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.593736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.596000 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.595977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f55ab0b2-0a83-4a87-9649-f7e71ffea294-kubelet-config\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.596093 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f55ab0b2-0a83-4a87-9649-f7e71ffea294-dbus\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.596093 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596033 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.596093 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:51.596253 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596047 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysconfig\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596253 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.596143 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:21:51.596253 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596171 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-run\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596253 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-var-lib-kubelet\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596253 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysctl-conf\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596253 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596251 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-systemd\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596270 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cnibin\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596283 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysctl-d\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596307 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-host\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596345 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-kubernetes\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-sys\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596417 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b708fbb9-e975-4422-90da-f2d2f53532d4-tmp\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-modprobe-d\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gr8c\" (UniqueName: \"kubernetes.io/projected/b708fbb9-e975-4422-90da-f2d2f53532d4-kube-api-access-5gr8c\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.596531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596507 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22cb6695-c93f-40b6-8af9-65d3988a2dee-agent-certs\") pod \"konnectivity-agent-lzkvk\" (UID: \"22cb6695-c93f-40b6-8af9-65d3988a2dee\") " pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596560 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596586 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5f5z\" (UniqueName: \"kubernetes.io/projected/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-kube-api-access-n5f5z\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596632 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-lib-modules\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-tuned\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-os-release\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596743 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22cb6695-c93f-40b6-8af9-65d3988a2dee-konnectivity-ca\") pod \"konnectivity-agent-lzkvk\" (UID: \"22cb6695-c93f-40b6-8af9-65d3988a2dee\") " pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-system-cni-dir\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596849 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.597019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.596876 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.598917 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.598227 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.598917 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.598279 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.598917 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.598448 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.599569 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.599532 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 18:21:51.599883 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.599868 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 18:21:51.600677 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.600538 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-98sh2\"" Jul 14 18:21:51.600677 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.600651 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgp7r\"" Jul 14 18:21:51.600971 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.600953 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 18:21:51.601248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.601234 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.601322 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.598230 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.601407 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.601395 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 18:21:51.601776 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.601743 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-62f7s\"" Jul 14 18:21:51.601973 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.601947 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 18:21:51.602093 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.602071 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.602282 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.602262 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.604631 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.604506 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:51.604631 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.604586 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:21:51.606708 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.606688 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.609195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.609175 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 18:21:51.609488 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.609469 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6xhjz\"" Jul 14 18:21:51.609698 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.609682 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.609922 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.609883 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 18:21:51.610168 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.610146 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cr9kq\"" Jul 14 18:21:51.610250 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.610205 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.610363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.610332 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 18:21:51.610629 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.610612 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 18:21:51.610749 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.610732 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnlpt\"" Jul 14 18:21:51.616899 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.616716 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:21:51.649207 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.649180 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dqmrh" Jul 14 18:21:51.657603 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.657586 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dqmrh" Jul 14 18:21:51.687910 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.687891 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 18:21:51.697580 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697555 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22cb6695-c93f-40b6-8af9-65d3988a2dee-konnectivity-ca\") pod \"konnectivity-agent-lzkvk\" (UID: \"22cb6695-c93f-40b6-8af9-65d3988a2dee\") " pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.697702 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-system-cni-dir\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.697702 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697622 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-etc-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.697702 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-socket-dir-parent\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.697702 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697673 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-cni-multus\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.697702 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697698 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f55ab0b2-0a83-4a87-9649-f7e71ffea294-kubelet-config\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysconfig\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-var-lib-kubelet\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cnibin\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697855 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqfv\" (UniqueName: \"kubernetes.io/projected/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-kube-api-access-9mqfv\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-systemd\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697901 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jds7x\" (UniqueName: \"kubernetes.io/projected/e896dc95-a4d2-4089-b398-4f226cc03794-kube-api-access-jds7x\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697925 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-systemd\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697947 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.697985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.697996 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-os-release\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-hostroot\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2b03911-b95b-46cb-bfca-20132d7d34a8-hosts-file\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698067 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458sf\" (UniqueName: \"kubernetes.io/projected/fc9a1bc7-930e-473c-b344-01a70ba50626-kube-api-access-458sf\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698092 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-sys\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcbafb7f-bcf0-485d-84f9-83783eef701f-host\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698166 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-systemd\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698175 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/22cb6695-c93f-40b6-8af9-65d3988a2dee-konnectivity-ca\") pod \"konnectivity-agent-lzkvk\" (UID: \"22cb6695-c93f-40b6-8af9-65d3988a2dee\") " pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698222 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f55ab0b2-0a83-4a87-9649-f7e71ffea294-kubelet-config\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698170 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-cni-bin\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysconfig\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-system-cni-dir\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698293 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cnibin\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698309 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-var-lib-kubelet\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698326 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-sys\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-host-slash\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.698386 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:51.698430 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-cnibin\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.698465 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:52.198445162 +0000 UTC m=+2.923585227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698487 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-cni-bin\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698525 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktgsh\" (UniqueName: \"kubernetes.io/projected/c2b03911-b95b-46cb-bfca-20132d7d34a8-kube-api-access-ktgsh\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698564 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22cb6695-c93f-40b6-8af9-65d3988a2dee-agent-certs\") pod \"konnectivity-agent-lzkvk\" (UID: \"22cb6695-c93f-40b6-8af9-65d3988a2dee\") " pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698601 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698685 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcbafb7f-bcf0-485d-84f9-83783eef701f-serviceca\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698717 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-env-overrides\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698735 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698751 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-lib-modules\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698799 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-sys-fs\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698829 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b03911-b95b-46cb-bfca-20132d7d34a8-tmp-dir\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-kubelet\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698880 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-multus-certs\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698908 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.699148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698913 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698933 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698963 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-slash\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.698986 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-run-netns\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-cni-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699030 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f55ab0b2-0a83-4a87-9649-f7e71ffea294-dbus\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-run\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-var-lib-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699154 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysctl-conf\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699180 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-device-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699223 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-ovn\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699247 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-cni-netd\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-daemon-config\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysctl-d\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-host\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699378 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:51.699804 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699406 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-node-log\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f55ab0b2-0a83-4a87-9649-f7e71ffea294-dbus\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699561 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysctl-conf\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-host\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699644 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-lib-modules\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699689 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-sysctl-d\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699834 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.699882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-run\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-system-cni-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700093 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-netns\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-cni-binary-copy\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700119 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-kubernetes\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b708fbb9-e975-4422-90da-f2d2f53532d4-tmp\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-socket-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-kubernetes\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700196 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-registration-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.700321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovnkube-config\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700250 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdlqh\" (UniqueName: \"kubernetes.io/projected/bb59207a-e4c3-4621-bf36-45f4c93d71e6-kube-api-access-bdlqh\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700271 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-k8s-cni-cncf-io\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-kubelet\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-modprobe-d\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gr8c\" (UniqueName: \"kubernetes.io/projected/b708fbb9-e975-4422-90da-f2d2f53532d4-kube-api-access-5gr8c\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700363 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5f5z\" (UniqueName: \"kubernetes.io/projected/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-kube-api-access-n5f5z\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn8rk\" (UniqueName: \"kubernetes.io/projected/fcbafb7f-bcf0-485d-84f9-83783eef701f-kube-api-access-zn8rk\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700409 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-log-socket\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovn-node-metrics-cert\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700486 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-etc-kubernetes\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700571 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-modprobe-d\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700709 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-tuned\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700777 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700803 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.700915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-iptables-alerter-script\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/816c5ab9-e120-4cb9-80cf-632f7cb990b4-cni-binary-copy\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700870 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-conf-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-os-release\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-systemd-units\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700938 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovnkube-script-lib\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.700962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k47j7\" (UniqueName: \"kubernetes.io/projected/816c5ab9-e120-4cb9-80cf-632f7cb990b4-kube-api-access-k47j7\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.701528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.701078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-os-release\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.704781 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.702853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/22cb6695-c93f-40b6-8af9-65d3988a2dee-agent-certs\") pod \"konnectivity-agent-lzkvk\" (UID: \"22cb6695-c93f-40b6-8af9-65d3988a2dee\") " pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.704781 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.703123 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b708fbb9-e975-4422-90da-f2d2f53532d4-etc-tuned\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.706305 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.706285 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b708fbb9-e975-4422-90da-f2d2f53532d4-tmp\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.711685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.711662 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5f5z\" (UniqueName: \"kubernetes.io/projected/a2d05a5a-3286-4a73-825d-6b0c1fbd6325-kube-api-access-n5f5z\") pod \"multus-additional-cni-plugins-k8lfb\" (UID: \"a2d05a5a-3286-4a73-825d-6b0c1fbd6325\") " pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.711965 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.711943 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gr8c\" (UniqueName: \"kubernetes.io/projected/b708fbb9-e975-4422-90da-f2d2f53532d4-kube-api-access-5gr8c\") pod \"tuned-xhwnr\" (UID: \"b708fbb9-e975-4422-90da-f2d2f53532d4\") " pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.782908 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.782880 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" event={"ID":"bfc6a64e8b5609e46d7d0fea51c9ccea","Type":"ContainerStarted","Data":"9af0a1103f2b335f950a9bfe3715d2accbb1ea9818057304af5b51f231f38ab3"} Jul 14 18:21:51.801619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.801588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:51.801738 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.801626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-node-log\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.802513 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.802002 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-node-log\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.802513 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.801651 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-system-cni-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.802669 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.802479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-system-cni-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.802740 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.802712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-netns\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.802859 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.802691 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-netns\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803022 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.802991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-socket-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.803070 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-registration-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.803121 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovnkube-config\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803166 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdlqh\" (UniqueName: \"kubernetes.io/projected/bb59207a-e4c3-4621-bf36-45f4c93d71e6-kube-api-access-bdlqh\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803210 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803186 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-k8s-cni-cncf-io\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803257 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-kubelet\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803299 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803257 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn8rk\" (UniqueName: \"kubernetes.io/projected/fcbafb7f-bcf0-485d-84f9-83783eef701f-kube-api-access-zn8rk\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.803351 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-log-socket\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803351 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovn-node-metrics-cert\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803442 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-etc-kubernetes\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803442 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803391 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.803576 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803629 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803587 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-iptables-alerter-script\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.803672 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803631 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/816c5ab9-e120-4cb9-80cf-632f7cb990b4-cni-binary-copy\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803672 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-conf-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803786 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-systemd-units\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803786 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovnkube-script-lib\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803786 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803748 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k47j7\" (UniqueName: \"kubernetes.io/projected/816c5ab9-e120-4cb9-80cf-632f7cb990b4-kube-api-access-k47j7\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803795 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-etc-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.803926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-socket-dir-parent\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.803926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803895 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-cni-multus\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.804056 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.803989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqfv\" (UniqueName: \"kubernetes.io/projected/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-kube-api-access-9mqfv\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.804105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jds7x\" (UniqueName: \"kubernetes.io/projected/e896dc95-a4d2-4089-b398-4f226cc03794-kube-api-access-jds7x\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.804150 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804083 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-systemd\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.804202 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804125 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.804255 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.804255 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804246 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-os-release\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.804399 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804384 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-hostroot\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.804447 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804410 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2b03911-b95b-46cb-bfca-20132d7d34a8-hosts-file\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.804447 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804438 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458sf\" (UniqueName: \"kubernetes.io/projected/fc9a1bc7-930e-473c-b344-01a70ba50626-kube-api-access-458sf\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:51.804532 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804480 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcbafb7f-bcf0-485d-84f9-83783eef701f-host\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.804578 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-cni-bin\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.804578 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-host-slash\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.804670 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-cnibin\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.804670 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-cni-bin\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.804784 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktgsh\" (UniqueName: \"kubernetes.io/projected/c2b03911-b95b-46cb-bfca-20132d7d34a8-kube-api-access-ktgsh\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.804837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.804882 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcbafb7f-bcf0-485d-84f9-83783eef701f-serviceca\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.804924 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.804885 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-env-overrides\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805087 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805064 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-cni-bin\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805087 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-host-slash\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.805179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805157 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-conf-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805158 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.805229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805120 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-os-release\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805314 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-etc-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805368 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805292 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-kubelet\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805419 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805381 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-socket-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.805419 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805404 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-registration-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.805520 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805454 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:51.805520 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-log-socket\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805520 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805490 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-sys-fs\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.805659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805525 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b03911-b95b-46cb-bfca-20132d7d34a8-tmp-dir\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.805659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805547 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-socket-dir-parent\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805562 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-kubelet\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805528 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-systemd-units\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805601 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-etc-kubernetes\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805624 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-multus-certs\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805674 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-slash\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805683 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fcbafb7f-bcf0-485d-84f9-83783eef701f-serviceca\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805688 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-var-lib-cni-multus\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805711 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-run-netns\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805746 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-run-netns\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805817 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-slash\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805853 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-cni-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805887 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-env-overrides\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.805944 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-var-lib-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805966 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c2b03911-b95b-46cb-bfca-20132d7d34a8-tmp-dir\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805979 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-device-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-run-ovn-kubernetes\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-ovn\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806029 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-device-dir\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.805992 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-cni-dir\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806078 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-var-lib-openvswitch\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806106 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-hostroot\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806318 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/816c5ab9-e120-4cb9-80cf-632f7cb990b4-cni-binary-copy\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806332 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c2b03911-b95b-46cb-bfca-20132d7d34a8-hosts-file\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.806411 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806364 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-kubelet\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806778 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.806322 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:51.806778 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.806577 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:52.306552108 +0000 UTC m=+3.031692159 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:51.806778 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-sys-fs\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.806886 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806812 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovnkube-script-lib\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806886 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806855 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovnkube-config\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806924 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-systemd\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.806947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.806933 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e896dc95-a4d2-4089-b398-4f226cc03794-etc-selinux\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.807027 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807007 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-multus-certs\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.807220 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807202 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-cnibin\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.807353 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-cni-netd\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.807463 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807418 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-cni-bin\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.807574 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807522 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-run-ovn\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.807922 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807867 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/816c5ab9-e120-4cb9-80cf-632f7cb990b4-host-run-k8s-cni-cncf-io\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.807922 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807866 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fcbafb7f-bcf0-485d-84f9-83783eef701f-host\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.807922 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807868 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-cni-netd\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.808085 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.807923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-daemon-config\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.808085 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.808021 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bb59207a-e4c3-4621-bf36-45f4c93d71e6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.808339 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.808317 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-iptables-alerter-script\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.808379 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.808344 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/816c5ab9-e120-4cb9-80cf-632f7cb990b4-multus-daemon-config\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.808981 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.808961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bb59207a-e4c3-4621-bf36-45f4c93d71e6-ovn-node-metrics-cert\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.828743 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.828684 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:51.828743 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.828711 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:51.828743 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.828726 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:51.828956 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:51.828818 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:52.328801008 +0000 UTC m=+3.053941061 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:51.831991 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.831967 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdlqh\" (UniqueName: \"kubernetes.io/projected/bb59207a-e4c3-4621-bf36-45f4c93d71e6-kube-api-access-bdlqh\") pod \"ovnkube-node-9hs2p\" (UID: \"bb59207a-e4c3-4621-bf36-45f4c93d71e6\") " pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.832077 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.832016 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jds7x\" (UniqueName: \"kubernetes.io/projected/e896dc95-a4d2-4089-b398-4f226cc03794-kube-api-access-jds7x\") pod \"aws-ebs-csi-driver-node-vmrlx\" (UID: \"e896dc95-a4d2-4089-b398-4f226cc03794\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.832136 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.832084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k47j7\" (UniqueName: \"kubernetes.io/projected/816c5ab9-e120-4cb9-80cf-632f7cb990b4-kube-api-access-k47j7\") pod \"multus-lxtcf\" (UID: \"816c5ab9-e120-4cb9-80cf-632f7cb990b4\") " pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.832176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.832162 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqfv\" (UniqueName: \"kubernetes.io/projected/b3aeca0d-66bc-4f6b-95ad-2eece3509fee-kube-api-access-9mqfv\") pod \"iptables-alerter-9j4j5\" (UID: \"b3aeca0d-66bc-4f6b-95ad-2eece3509fee\") " pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.832492 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.832471 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn8rk\" (UniqueName: \"kubernetes.io/projected/fcbafb7f-bcf0-485d-84f9-83783eef701f-kube-api-access-zn8rk\") pod \"node-ca-v9k87\" (UID: \"fcbafb7f-bcf0-485d-84f9-83783eef701f\") " pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.833012 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.832994 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktgsh\" (UniqueName: \"kubernetes.io/projected/c2b03911-b95b-46cb-bfca-20132d7d34a8-kube-api-access-ktgsh\") pod \"node-resolver-6fzdg\" (UID: \"c2b03911-b95b-46cb-bfca-20132d7d34a8\") " pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:51.834436 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.834413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-458sf\" (UniqueName: \"kubernetes.io/projected/fc9a1bc7-930e-473c-b344-01a70ba50626-kube-api-access-458sf\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:51.897170 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.897141 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" Jul 14 18:21:51.907685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.907661 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:21:51.909126 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.909107 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:51.916348 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.916329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" Jul 14 18:21:51.921988 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.921968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" Jul 14 18:21:51.929819 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.929803 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-v9k87" Jul 14 18:21:51.938583 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.938562 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:21:51.945169 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.945151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lxtcf" Jul 14 18:21:51.952631 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.952613 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9j4j5" Jul 14 18:21:51.958266 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:51.958249 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6fzdg" Jul 14 18:21:52.180564 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.180534 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b03911_b95b_46cb_bfca_20132d7d34a8.slice/crio-b154df7e39cd33f5e7b4c2eb83558adb0c67a1ce07ef656130e7dddd97a28076 WatchSource:0}: Error finding container b154df7e39cd33f5e7b4c2eb83558adb0c67a1ce07ef656130e7dddd97a28076: Status 404 returned error can't find the container with id b154df7e39cd33f5e7b4c2eb83558adb0c67a1ce07ef656130e7dddd97a28076 Jul 14 18:21:52.181823 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.181747 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cb6695_c93f_40b6_8af9_65d3988a2dee.slice/crio-5cfa3de88d4f6c29244dbcc2ac61d3077947a6cc2de1060ffae8e343015aeaa5 WatchSource:0}: Error finding container 5cfa3de88d4f6c29244dbcc2ac61d3077947a6cc2de1060ffae8e343015aeaa5: Status 404 returned error can't find the container with id 5cfa3de88d4f6c29244dbcc2ac61d3077947a6cc2de1060ffae8e343015aeaa5 Jul 14 18:21:52.185588 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.185510 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d05a5a_3286_4a73_825d_6b0c1fbd6325.slice/crio-dcf73b07ef61211184ffbf20be0697d86e16715900cebb0846308ed0ee023c5e WatchSource:0}: Error finding container dcf73b07ef61211184ffbf20be0697d86e16715900cebb0846308ed0ee023c5e: Status 404 returned error can't find the container with id dcf73b07ef61211184ffbf20be0697d86e16715900cebb0846308ed0ee023c5e Jul 14 18:21:52.189576 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.187964 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3aeca0d_66bc_4f6b_95ad_2eece3509fee.slice/crio-cd0be25fcded1538a782f8ae0d28e851e554693705ff7b1c5283467c3d4846b5 WatchSource:0}: Error finding container cd0be25fcded1538a782f8ae0d28e851e554693705ff7b1c5283467c3d4846b5: Status 404 returned error can't find the container with id cd0be25fcded1538a782f8ae0d28e851e554693705ff7b1c5283467c3d4846b5 Jul 14 18:21:52.189576 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.188412 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb708fbb9_e975_4422_90da_f2d2f53532d4.slice/crio-61c62630034209bd4d9511320a495f921e41fa3f979ddede4e35545ab19a1377 WatchSource:0}: Error finding container 61c62630034209bd4d9511320a495f921e41fa3f979ddede4e35545ab19a1377: Status 404 returned error can't find the container with id 61c62630034209bd4d9511320a495f921e41fa3f979ddede4e35545ab19a1377 Jul 14 18:21:52.189576 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.188970 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb59207a_e4c3_4621_bf36_45f4c93d71e6.slice/crio-dfb895f1ef9710cc5bbd7c4f140c97a44f1d5e254f600754fe6b9a16d0bbbcaf WatchSource:0}: Error finding container dfb895f1ef9710cc5bbd7c4f140c97a44f1d5e254f600754fe6b9a16d0bbbcaf: Status 404 returned error can't find the container with id dfb895f1ef9710cc5bbd7c4f140c97a44f1d5e254f600754fe6b9a16d0bbbcaf Jul 14 18:21:52.210427 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.210405 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:52.210671 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.210509 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:52.210671 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.210569 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:53.210552236 +0000 UTC m=+3.935692277 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:52.211094 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:21:52.211076 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcbafb7f_bcf0_485d_84f9_83783eef701f.slice/crio-e161e44895d173cf5f8a1a11ffcf132c1b6dab71b7e70aa89d0269d32a5e1246 WatchSource:0}: Error finding container e161e44895d173cf5f8a1a11ffcf132c1b6dab71b7e70aa89d0269d32a5e1246: Status 404 returned error can't find the container with id e161e44895d173cf5f8a1a11ffcf132c1b6dab71b7e70aa89d0269d32a5e1246 Jul 14 18:21:52.311810 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.311615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:52.311962 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.311742 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:52.311962 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.311883 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:53.311866012 +0000 UTC m=+4.037006052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:52.412701 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.412677 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:52.412842 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.412827 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:52.412903 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.412848 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:52.412903 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.412860 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:52.412989 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.412916 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:53.412895756 +0000 UTC m=+4.138035810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:52.661105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.661039 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:16:51 +0000 UTC" deadline="2028-01-04 11:42:50.222212191 +0000 UTC" Jul 14 18:21:52.661105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.661103 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12929h20m57.561113545s" Jul 14 18:21:52.778774 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.777911 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:52.778774 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.778040 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:21:52.778774 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.778435 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:52.778774 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.778525 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:21:52.778774 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.778590 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:52.778774 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:52.778670 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:21:52.797959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.797927 2580 generic.go:356] "Generic (PLEG): container finished" podID="bfc6a64e8b5609e46d7d0fea51c9ccea" containerID="cb99bb7aba7f48e5c54b7560734fb65a22ed2447c373819e945cf0cc960a3e31" exitCode=0 Jul 14 18:21:52.798106 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.798018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" event={"ID":"bfc6a64e8b5609e46d7d0fea51c9ccea","Type":"ContainerDied","Data":"cb99bb7aba7f48e5c54b7560734fb65a22ed2447c373819e945cf0cc960a3e31"} Jul 14 18:21:52.805889 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.805819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9k87" event={"ID":"fcbafb7f-bcf0-485d-84f9-83783eef701f","Type":"ContainerStarted","Data":"e161e44895d173cf5f8a1a11ffcf132c1b6dab71b7e70aa89d0269d32a5e1246"} Jul 14 18:21:52.810071 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.809971 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"dfb895f1ef9710cc5bbd7c4f140c97a44f1d5e254f600754fe6b9a16d0bbbcaf"} Jul 14 18:21:52.820719 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.820483 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" event={"ID":"b708fbb9-e975-4422-90da-f2d2f53532d4","Type":"ContainerStarted","Data":"61c62630034209bd4d9511320a495f921e41fa3f979ddede4e35545ab19a1377"} Jul 14 18:21:52.832668 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.832584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9j4j5" event={"ID":"b3aeca0d-66bc-4f6b-95ad-2eece3509fee","Type":"ContainerStarted","Data":"cd0be25fcded1538a782f8ae0d28e851e554693705ff7b1c5283467c3d4846b5"} Jul 14 18:21:52.847055 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.846984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxtcf" event={"ID":"816c5ab9-e120-4cb9-80cf-632f7cb990b4","Type":"ContainerStarted","Data":"55162ee424deef3a996123fb1d903ab52f6e733c63eb38968762589f906f5ea3"} Jul 14 18:21:52.851593 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.851565 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzkvk" event={"ID":"22cb6695-c93f-40b6-8af9-65d3988a2dee","Type":"ContainerStarted","Data":"5cfa3de88d4f6c29244dbcc2ac61d3077947a6cc2de1060ffae8e343015aeaa5"} Jul 14 18:21:52.860570 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.860500 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" event={"ID":"e896dc95-a4d2-4089-b398-4f226cc03794","Type":"ContainerStarted","Data":"dd53cb328348150d79b07bdfd43e518ad9dcc01288dcfd6ae3c5907a6e031c8d"} Jul 14 18:21:52.883007 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.882947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerStarted","Data":"dcf73b07ef61211184ffbf20be0697d86e16715900cebb0846308ed0ee023c5e"} Jul 14 18:21:52.889672 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:52.889608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6fzdg" event={"ID":"c2b03911-b95b-46cb-bfca-20132d7d34a8","Type":"ContainerStarted","Data":"b154df7e39cd33f5e7b4c2eb83558adb0c67a1ce07ef656130e7dddd97a28076"} Jul 14 18:21:53.218318 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.218282 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:53.218493 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.218455 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:53.218562 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.218513 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:55.218495593 +0000 UTC m=+5.943635647 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:53.318726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.318646 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:53.318916 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.318824 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:53.318916 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.318894 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:55.31887334 +0000 UTC m=+6.044013393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:53.419801 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.419750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:53.419978 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.419961 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:53.420043 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.419980 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:53.420043 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.419993 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:53.420145 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:53.420049 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:55.420030187 +0000 UTC m=+6.145170243 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:53.584162 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.584080 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:53.661964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.661916 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:16:51 +0000 UTC" deadline="2028-02-20 13:52:21.903418575 +0000 UTC" Jul 14 18:21:53.661964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.661952 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14059h30m28.241470347s" Jul 14 18:21:53.907833 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.907300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" event={"ID":"bfc6a64e8b5609e46d7d0fea51c9ccea","Type":"ContainerStarted","Data":"ad7bc9b17dfdc05d3667f18d58f5e9985fcf1c08dd1f8d5bd950addd0e9680d5"} Jul 14 18:21:53.933320 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:53.932815 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-199.ec2.internal" podStartSLOduration=3.932795044 podStartE2EDuration="3.932795044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:21:53.932097555 +0000 UTC m=+4.657237617" watchObservedRunningTime="2026-07-14 18:21:53.932795044 +0000 UTC m=+4.657935107" Jul 14 18:21:54.777858 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:54.777823 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:54.778324 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:54.777962 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:21:54.778392 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:54.778370 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:54.778493 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:54.778470 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:21:54.778559 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:54.778524 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:54.778613 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:54.778578 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:21:55.235533 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:55.235452 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:55.235688 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.235577 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:55.235688 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.235650 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:59.235629895 +0000 UTC m=+9.960769940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:55.336488 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:55.336044 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:55.336488 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.336177 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:55.336488 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.336233 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:59.336214289 +0000 UTC m=+10.061354350 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:55.436471 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:55.436434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:55.436636 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.436586 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:55.436636 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.436609 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:55.436636 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.436622 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:55.436821 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:55.436675 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:59.436656489 +0000 UTC m=+10.161796544 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:56.778315 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:56.778284 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:56.778776 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:56.778390 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:21:56.778776 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:56.778712 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:56.778900 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:56.778791 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:21:56.778900 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:56.778857 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:56.779006 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:56.778915 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:21:58.777967 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:58.777932 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:58.778412 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:58.778061 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:21:58.778412 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:58.778130 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:58.778412 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:58.778206 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:21:58.778412 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:58.778245 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:58.778412 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:58.778300 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:21:59.270157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:59.270080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:21:59.270293 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.270206 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:59.270293 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.270269 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:07.270255368 +0000 UTC m=+17.995395407 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:59.371468 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:59.370912 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:21:59.371468 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.371087 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:59.371468 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.371148 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:07.371127785 +0000 UTC m=+18.096267832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:59.471920 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:21:59.471879 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:21:59.472101 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.472042 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:59.472101 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.472059 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:59.472101 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.472072 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:59.472257 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:21:59.472132 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:07.472110252 +0000 UTC m=+18.197250315 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:00.777822 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:00.777789 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:00.778378 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:00.777839 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:00.778378 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:00.777925 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:00.778378 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:00.778004 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:00.778378 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:00.777996 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:00.778378 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:00.778082 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:02.778048 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:02.778015 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:02.778441 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:02.778123 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:02.778441 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:02.778130 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:02.778441 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:02.778221 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:02.778441 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:02.778128 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:02.778441 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:02.778316 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:04.778011 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:04.777976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:04.778505 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:04.777976 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:04.778505 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:04.778107 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:04.778505 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:04.778196 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:04.778505 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:04.777988 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:04.778505 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:04.778289 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:06.778265 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:06.778226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:06.778755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:06.778226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:06.778755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:06.778364 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:06.778755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:06.778371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:06.778755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:06.778462 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:06.778755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:06.778566 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:07.332733 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:07.332707 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:07.332871 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.332835 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:07.332917 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.332887 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.332873329 +0000 UTC m=+34.058013368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:07.433623 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:07.433548 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:07.433752 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.433646 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:07.433752 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.433694 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.433680064 +0000 UTC m=+34.158820103 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:07.534197 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:07.534158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:07.534353 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.534329 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:22:07.534406 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.534363 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:22:07.534406 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.534376 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:07.534484 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:07.534428 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:23.534413141 +0000 UTC m=+34.259553190 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:08.777540 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:08.777506 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:08.777981 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:08.777508 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:08.777981 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:08.777627 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:08.777981 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:08.777638 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:08.777981 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:08.777739 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:08.777981 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:08.777868 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:10.778231 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:10.778200 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:10.778671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:10.778264 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:10.778671 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:10.778403 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:10.778785 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:10.778776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:10.778893 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:10.778873 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:10.778981 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:10.778958 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:11.946218 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:11.946153 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lzkvk" event={"ID":"22cb6695-c93f-40b6-8af9-65d3988a2dee","Type":"ContainerStarted","Data":"8ec172827c6f6e6dc0e0cd155c9781f9be88d58f5013d12b5db672bf22e841f6"} Jul 14 18:22:11.947545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:11.947524 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" event={"ID":"e896dc95-a4d2-4089-b398-4f226cc03794","Type":"ContainerStarted","Data":"0c71ae7feb44aeaefbd6f24d1e0078b40095a90f9293b2ff2ef033232181eed9"} Jul 14 18:22:11.983305 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:11.983252 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lzkvk" podStartSLOduration=7.762914287 podStartE2EDuration="22.98323315s" totalImagesPullingTime="15.220318863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:11.982447312 +0000 UTC m=+22.707587364" watchObservedRunningTime="2026-07-14 18:22:11.98323315 +0000 UTC m=+22.708373212" Jul 14 18:22:12.650663 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.650354 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:22:12.650915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.650900 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:22:12.777509 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.777478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:12.777654 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.777585 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:12.777654 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.777616 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:12.777780 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:12.777584 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:12.777780 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:12.777704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:12.777852 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:12.777810 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:12.951246 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.951209 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lxtcf" event={"ID":"816c5ab9-e120-4cb9-80cf-632f7cb990b4","Type":"ContainerStarted","Data":"cb067280f86fe50a4d79a5b9bcc367718622ab3b70a7bacb787ba3d223273c63"} Jul 14 18:22:12.953132 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.953105 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d05a5a-3286-4a73-825d-6b0c1fbd6325" containerID="6afc6dc45d806179493120f19e7e1d6af99f7fb6af0b93f8f10ca597c176816f" exitCode=0 Jul 14 18:22:12.953254 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.953198 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerDied","Data":"6afc6dc45d806179493120f19e7e1d6af99f7fb6af0b93f8f10ca597c176816f"} Jul 14 18:22:12.954853 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.954791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6fzdg" event={"ID":"c2b03911-b95b-46cb-bfca-20132d7d34a8","Type":"ContainerStarted","Data":"2051d4739618ad2652fd7a34a237e498942fddc8fd28a79ec13aa94731ce6355"} Jul 14 18:22:12.963331 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.963306 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" event={"ID":"eaac1d1135f2c4810bad5bfddca8e38b","Type":"ContainerStarted","Data":"8bed9aa4f73f85ca15986ea91158577e12d2c80c51c5b59f609633e3bee2aeb1"} Jul 14 18:22:12.964949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.964919 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-v9k87" event={"ID":"fcbafb7f-bcf0-485d-84f9-83783eef701f","Type":"ContainerStarted","Data":"7030912d4867d55b194b5167a50f9b0cc391df53c0f8d137e74efa729ff134c2"} Jul 14 18:22:12.968105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968086 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:22:12.968455 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968429 2580 generic.go:356] "Generic (PLEG): container finished" podID="bb59207a-e4c3-4621-bf36-45f4c93d71e6" containerID="f483171a765826e512f06241cb4bb67e6d622f3b84fdd89b1640791fea523294" exitCode=1 Jul 14 18:22:12.968547 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968504 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"27ae9fb72e7e31b81d18d2f22485f00bd5c7f4577179c42a8db28c9b8550ba8d"} Jul 14 18:22:12.968611 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968579 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"a7ac8f6ec89cbb6dd24e25cd631759cf40258dc93b68044dfc54219a48c74ad8"} Jul 14 18:22:12.968667 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968627 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"7719a03e6b326df7437f0a5f86d0fd07fc3c98c4ccacff579d41864be2eec242"} Jul 14 18:22:12.968724 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"bca62e17a77bded5e81c8e349f7ca6755a7982023cee320707557bd483291a25"} Jul 14 18:22:12.968799 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerDied","Data":"f483171a765826e512f06241cb4bb67e6d622f3b84fdd89b1640791fea523294"} Jul 14 18:22:12.968799 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.968792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"90a621a0d28f33440c9cccbff48c36678383f13526c8751008efedf5b68196e7"} Jul 14 18:22:12.970115 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.970085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" event={"ID":"b708fbb9-e975-4422-90da-f2d2f53532d4","Type":"ContainerStarted","Data":"8fceefb407c0c5e497e6c976a6f809eae75ede2e4e698d917f02c670a691ea1c"} Jul 14 18:22:12.975260 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.975217 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lxtcf" podStartSLOduration=4.042704054 podStartE2EDuration="23.97520558s" totalImagesPullingTime="19.932501526s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:12.974548991 +0000 UTC m=+23.699689051" watchObservedRunningTime="2026-07-14 18:22:12.97520558 +0000 UTC m=+23.700345640" Jul 14 18:22:12.991203 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:12.991157 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6fzdg" podStartSLOduration=4.478567307 podStartE2EDuration="23.991146946s" totalImagesPullingTime="19.512579639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:12.990944263 +0000 UTC m=+23.716084326" watchObservedRunningTime="2026-07-14 18:22:12.991146946 +0000 UTC m=+23.716287007" Jul 14 18:22:13.007661 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.007621 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-199.ec2.internal" podStartSLOduration=23.00760337 podStartE2EDuration="23.00760337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:13.007585981 +0000 UTC m=+23.732726042" watchObservedRunningTime="2026-07-14 18:22:13.00760337 +0000 UTC m=+23.732743432" Jul 14 18:22:13.031756 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.031722 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xhwnr" podStartSLOduration=4.55020715 podStartE2EDuration="24.031710086s" totalImagesPullingTime="19.481502936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:13.031438311 +0000 UTC m=+23.756578377" watchObservedRunningTime="2026-07-14 18:22:13.031710086 +0000 UTC m=+23.756850146" Jul 14 18:22:13.602940 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.602911 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 18:22:13.657964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.657797 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:22:13.602927934Z","UUID":"9b1873ea-d230-46d4-a55a-e28fab5c89a1","Handler":null,"Name":"","Endpoint":""} Jul 14 18:22:13.659154 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.659136 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 18:22:13.659270 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.659170 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 18:22:13.973923 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.973834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9j4j5" event={"ID":"b3aeca0d-66bc-4f6b-95ad-2eece3509fee","Type":"ContainerStarted","Data":"513652199ad851925161747ad9642c548f6f3f4ad15928305713687208a2abc4"} Jul 14 18:22:13.975872 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.975844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" event={"ID":"e896dc95-a4d2-4089-b398-4f226cc03794","Type":"ContainerStarted","Data":"f814bf169f2feb75d320a8ede914b2f3a0998bc61d4c5fb745cd8c6a96c6fe08"} Jul 14 18:22:13.976222 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.976208 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 18:22:13.998696 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.998646 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-v9k87" podStartSLOduration=5.521154834 podStartE2EDuration="24.998628019s" totalImagesPullingTime="19.477473185s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:13.094089965 +0000 UTC m=+23.819230026" watchObservedRunningTime="2026-07-14 18:22:13.998628019 +0000 UTC m=+24.723768079" Jul 14 18:22:13.999231 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:13.999204 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9j4j5" podStartSLOduration=5.518774631 podStartE2EDuration="24.999197491s" totalImagesPullingTime="19.48042286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:13.998771316 +0000 UTC m=+24.723911359" watchObservedRunningTime="2026-07-14 18:22:13.999197491 +0000 UTC m=+24.724337559" Jul 14 18:22:14.777352 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:14.777329 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:14.777467 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:14.777363 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:14.777568 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:14.777457 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:14.777568 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:14.777498 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:14.777568 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:14.777559 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:14.777693 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:14.777666 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:14.982171 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:14.981969 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:22:14.983051 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:14.983025 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"fe83d82f7867321439ff2082e03a73fdf352e1fa249a5109be100deb73a9a9d4"} Jul 14 18:22:16.246502 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:16.246466 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:22:16.247107 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:16.246600 2580 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 18:22:16.247308 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:16.247272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lzkvk" Jul 14 18:22:16.778285 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:16.778256 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:16.778285 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:16.778281 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:16.778523 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:16.778289 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:16.778523 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:16.778380 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:16.778523 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:16.778471 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:16.778671 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:16.778537 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:17.993510 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.993344 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:22:17.994037 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.993950 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"8f646957799facde165b99449fb176c012844006324ac4e743b1e30213dff04c"} Jul 14 18:22:17.994502 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.994464 2580 scope.go:117] "RemoveContainer" containerID="f483171a765826e512f06241cb4bb67e6d622f3b84fdd89b1640791fea523294" Jul 14 18:22:17.994820 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.994796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:22:17.994913 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.994875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:22:17.994962 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.994919 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:22:17.996266 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:17.996239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerStarted","Data":"345d80f2dab51a98a010ae05ecb867b6e5b3892c82e8f22bddc30042713de0ba"} Jul 14 18:22:18.011331 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:18.011307 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:22:18.012317 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:18.012301 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:22:18.778238 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:18.778211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:18.778342 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:18.778211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:18.778342 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:18.778320 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:18.778428 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:18.778211 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:18.778428 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:18.778411 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:18.778499 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:18.778455 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:19.001909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.001841 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:22:19.002359 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.002225 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" event={"ID":"bb59207a-e4c3-4621-bf36-45f4c93d71e6","Type":"ContainerStarted","Data":"bcbc611a218da3edf85c8aecd32da4849b190abc9b9ec66515bc6d8ebc874f62"} Jul 14 18:22:19.004341 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.004307 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" event={"ID":"e896dc95-a4d2-4089-b398-4f226cc03794","Type":"ContainerStarted","Data":"fe1f76d2de78a04cda0241db4b8449ef16d29ecbdd929221e7c790b160ecfc9f"} Jul 14 18:22:19.005988 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.005963 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d05a5a-3286-4a73-825d-6b0c1fbd6325" containerID="345d80f2dab51a98a010ae05ecb867b6e5b3892c82e8f22bddc30042713de0ba" exitCode=0 Jul 14 18:22:19.006095 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.006006 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerDied","Data":"345d80f2dab51a98a010ae05ecb867b6e5b3892c82e8f22bddc30042713de0ba"} Jul 14 18:22:19.169998 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.169952 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" podStartSLOduration=10.334160671 podStartE2EDuration="30.169937664s" totalImagesPullingTime="19.835776993s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:19.123935863 +0000 UTC m=+29.849075925" watchObservedRunningTime="2026-07-14 18:22:19.169937664 +0000 UTC m=+29.895077724" Jul 14 18:22:19.200154 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:19.200112 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vmrlx" podStartSLOduration=4.149906044 podStartE2EDuration="30.200097905s" totalImagesPullingTime="26.050191861s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:19.171900169 +0000 UTC m=+29.897040230" watchObservedRunningTime="2026-07-14 18:22:19.200097905 +0000 UTC m=+29.925237944" Jul 14 18:22:20.024722 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:20.024686 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5wktq"] Jul 14 18:22:20.025137 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:20.024845 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:20.025137 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:20.024964 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:20.029928 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:20.029906 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65qbb"] Jul 14 18:22:20.030067 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:20.029996 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:20.030130 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:20.030094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:20.030410 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:20.030393 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wc7l6"] Jul 14 18:22:20.030482 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:20.030470 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:20.030585 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:20.030564 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:21.778422 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:21.778397 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:21.778833 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:21.778406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:21.778833 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:21.778495 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:21.778833 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:21.778406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:21.778833 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:21.778593 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:21.778833 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:21.778686 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:23.017817 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.017789 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d05a5a-3286-4a73-825d-6b0c1fbd6325" containerID="91f02554f75bc9d509378a68aeeb46aedcf564e3988d62f0b7acb8c3e43ab30d" exitCode=0 Jul 14 18:22:23.018308 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.017831 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerDied","Data":"91f02554f75bc9d509378a68aeeb46aedcf564e3988d62f0b7acb8c3e43ab30d"} Jul 14 18:22:23.360264 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.360228 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:23.360421 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.360335 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:23.360421 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.360387 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret podName:f55ab0b2-0a83-4a87-9649-f7e71ffea294 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:55.360372474 +0000 UTC m=+66.085512515 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret") pod "global-pull-secret-syncer-65qbb" (UID: "f55ab0b2-0a83-4a87-9649-f7e71ffea294") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:23.461570 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.461540 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:23.461727 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.461669 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:23.461727 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.461726 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs podName:fc9a1bc7-930e-473c-b344-01a70ba50626 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:55.461711763 +0000 UTC m=+66.186851802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs") pod "network-metrics-daemon-5wktq" (UID: "fc9a1bc7-930e-473c-b344-01a70ba50626") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:23.562774 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.562723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:23.562926 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.562901 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:22:23.562926 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.562923 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:22:23.562990 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.562934 2580 projected.go:196] Error preparing data for projected volume kube-api-access-gjfqr for pod openshift-network-diagnostics/network-check-target-wc7l6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:23.563022 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.562990 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr podName:845e9932-2592-4385-b8a2-229a236a8700 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:55.56297497 +0000 UTC m=+66.288115014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gjfqr" (UniqueName: "kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr") pod "network-check-target-wc7l6" (UID: "845e9932-2592-4385-b8a2-229a236a8700") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:23.777870 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.777844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:23.778020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.777844 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:23.778020 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.777949 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-65qbb" podUID="f55ab0b2-0a83-4a87-9649-f7e71ffea294" Jul 14 18:22:23.778107 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.778033 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5wktq" podUID="fc9a1bc7-930e-473c-b344-01a70ba50626" Jul 14 18:22:23.778107 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:23.777850 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:23.778188 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:23.778108 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wc7l6" podUID="845e9932-2592-4385-b8a2-229a236a8700" Jul 14 18:22:24.022429 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:24.022398 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d05a5a-3286-4a73-825d-6b0c1fbd6325" containerID="d7f3f2192adc3c3a033fa24cf8324bed2840106e640ad6264c1032fb13627957" exitCode=0 Jul 14 18:22:24.022963 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:24.022439 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerDied","Data":"d7f3f2192adc3c3a033fa24cf8324bed2840106e640ad6264c1032fb13627957"} Jul 14 18:22:25.619076 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.619039 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-199.ec2.internal" event="NodeReady" Jul 14 18:22:25.619491 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.619187 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 18:22:25.672318 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.672279 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jzc5q"] Jul 14 18:22:25.675942 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.675910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.678380 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.678349 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-stjst"] Jul 14 18:22:25.679337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.679314 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-58lnk\"" Jul 14 18:22:25.679616 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.679591 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 18:22:25.679718 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.679699 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 18:22:25.681689 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.681668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.685352 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.685326 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 18:22:25.686626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.686602 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 18:22:25.686706 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.686651 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 18:22:25.686750 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.686651 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wk6vb\"" Jul 14 18:22:25.693041 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.693017 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jzc5q"] Jul 14 18:22:25.701874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.701848 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-stjst"] Jul 14 18:22:25.777656 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.777622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:25.777838 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.777625 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:25.777838 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.777775 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:25.778184 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.777636 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzh2r\" (UniqueName: \"kubernetes.io/projected/13730500-ea99-40e3-a88a-07a5593ac288-kube-api-access-rzh2r\") pod \"ingress-canary-jzc5q\" (UID: \"13730500-ea99-40e3-a88a-07a5593ac288\") " pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.778184 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.778183 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ae0b10-8390-49bd-844c-18be5fb5a09c-tmp-dir\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.778354 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.778211 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.778354 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.778235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81ae0b10-8390-49bd-844c-18be5fb5a09c-config-volume\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.778354 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.778284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk48s\" (UniqueName: \"kubernetes.io/projected/81ae0b10-8390-49bd-844c-18be5fb5a09c-kube-api-access-hk48s\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.778482 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.778380 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13730500-ea99-40e3-a88a-07a5593ac288-cert\") pod \"ingress-canary-jzc5q\" (UID: \"13730500-ea99-40e3-a88a-07a5593ac288\") " pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.781839 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.781611 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:22:25.781839 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.781619 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lzfqw\"" Jul 14 18:22:25.781839 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.781709 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzvpc\"" Jul 14 18:22:25.782032 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.781902 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:22:25.782032 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.781957 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:22:25.782315 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.782298 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:22:25.878750 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.878719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13730500-ea99-40e3-a88a-07a5593ac288-cert\") pod \"ingress-canary-jzc5q\" (UID: \"13730500-ea99-40e3-a88a-07a5593ac288\") " pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.878921 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.878775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzh2r\" (UniqueName: \"kubernetes.io/projected/13730500-ea99-40e3-a88a-07a5593ac288-kube-api-access-rzh2r\") pod \"ingress-canary-jzc5q\" (UID: \"13730500-ea99-40e3-a88a-07a5593ac288\") " pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.878921 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.878845 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ae0b10-8390-49bd-844c-18be5fb5a09c-tmp-dir\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.878921 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.878875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.878921 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.878902 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81ae0b10-8390-49bd-844c-18be5fb5a09c-config-volume\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.879111 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.878948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk48s\" (UniqueName: \"kubernetes.io/projected/81ae0b10-8390-49bd-844c-18be5fb5a09c-kube-api-access-hk48s\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.879298 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:25.879278 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:25.879374 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:25.879361 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls podName:81ae0b10-8390-49bd-844c-18be5fb5a09c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:26.379339084 +0000 UTC m=+37.104479125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls") pod "dns-default-stjst" (UID: "81ae0b10-8390-49bd-844c-18be5fb5a09c") : secret "dns-default-metrics-tls" not found Jul 14 18:22:25.879437 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.879358 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81ae0b10-8390-49bd-844c-18be5fb5a09c-tmp-dir\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.879831 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.879808 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/81ae0b10-8390-49bd-844c-18be5fb5a09c-config-volume\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.883980 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.883960 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/13730500-ea99-40e3-a88a-07a5593ac288-cert\") pod \"ingress-canary-jzc5q\" (UID: \"13730500-ea99-40e3-a88a-07a5593ac288\") " pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.896010 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.895985 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzh2r\" (UniqueName: \"kubernetes.io/projected/13730500-ea99-40e3-a88a-07a5593ac288-kube-api-access-rzh2r\") pod \"ingress-canary-jzc5q\" (UID: \"13730500-ea99-40e3-a88a-07a5593ac288\") " pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:25.896674 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.896649 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk48s\" (UniqueName: \"kubernetes.io/projected/81ae0b10-8390-49bd-844c-18be5fb5a09c-kube-api-access-hk48s\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:25.989710 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:25.989685 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jzc5q" Jul 14 18:22:26.145980 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:26.145689 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jzc5q"] Jul 14 18:22:26.151503 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:26.151477 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13730500_ea99_40e3_a88a_07a5593ac288.slice/crio-0aa17be10379e1f36438ebd4a87a4c693b4a3f90e5bbb504664e46d21abc1f50 WatchSource:0}: Error finding container 0aa17be10379e1f36438ebd4a87a4c693b4a3f90e5bbb504664e46d21abc1f50: Status 404 returned error can't find the container with id 0aa17be10379e1f36438ebd4a87a4c693b4a3f90e5bbb504664e46d21abc1f50 Jul 14 18:22:26.383152 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:26.383068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:26.383309 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:26.383236 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:26.383363 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:26.383318 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls podName:81ae0b10-8390-49bd-844c-18be5fb5a09c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.383292308 +0000 UTC m=+38.108432361 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls") pod "dns-default-stjst" (UID: "81ae0b10-8390-49bd-844c-18be5fb5a09c") : secret "dns-default-metrics-tls" not found Jul 14 18:22:27.030853 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.030812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jzc5q" event={"ID":"13730500-ea99-40e3-a88a-07a5593ac288","Type":"ContainerStarted","Data":"0aa17be10379e1f36438ebd4a87a4c693b4a3f90e5bbb504664e46d21abc1f50"} Jul 14 18:22:27.073789 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.073741 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2qs8b"] Jul 14 18:22:27.078473 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.078444 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.082234 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.082212 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 18:22:27.082234 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.082223 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtlhx\"" Jul 14 18:22:27.082420 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.082241 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 18:22:27.082420 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.082214 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 18:22:27.083733 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.083714 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 18:22:27.098625 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.098605 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qs8b"] Jul 14 18:22:27.189173 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.189138 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.189173 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.189175 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85330e72-de6f-4c21-a6b8-a912f561c69d-crio-socket\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.189377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.189200 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk5jv\" (UniqueName: \"kubernetes.io/projected/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-api-access-qk5jv\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.189377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.189221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85330e72-de6f-4c21-a6b8-a912f561c69d-data-volume\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.189377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.189238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85330e72-de6f-4c21-a6b8-a912f561c69d-crio-socket\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.290165 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.290261 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.790242855 +0000 UTC m=+38.515382919 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : configmap "kube-rbac-proxy" not found Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk5jv\" (UniqueName: \"kubernetes.io/projected/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-api-access-qk5jv\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290305 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85330e72-de6f-4c21-a6b8-a912f561c69d-crio-socket\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290316 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85330e72-de6f-4c21-a6b8-a912f561c69d-data-volume\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.290485 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.290526 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.790512086 +0000 UTC m=+38.515652127 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.290755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.290675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85330e72-de6f-4c21-a6b8-a912f561c69d-data-volume\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.306590 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.306562 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk5jv\" (UniqueName: \"kubernetes.io/projected/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-api-access-qk5jv\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.391666 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.391626 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:27.391890 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.391872 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:27.391964 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.391948 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls podName:81ae0b10-8390-49bd-844c-18be5fb5a09c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:29.391928819 +0000 UTC m=+40.117068864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls") pod "dns-default-stjst" (UID: "81ae0b10-8390-49bd-844c-18be5fb5a09c") : secret "dns-default-metrics-tls" not found Jul 14 18:22:27.790621 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.790588 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c"] Jul 14 18:22:27.793933 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.793907 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" Jul 14 18:22:27.795236 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.795212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.795371 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.795265 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:27.795427 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.795386 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:27.795427 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.795415 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.795523 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.795476 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:28.795454592 +0000 UTC m=+39.520594631 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : configmap "kube-rbac-proxy" not found Jul 14 18:22:27.795523 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:27.795506 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:28.795498964 +0000 UTC m=+39.520639002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.797140 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.797116 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 14 18:22:27.797532 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.797505 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 14 18:22:27.797715 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.797697 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-fqvmz\"" Jul 14 18:22:27.810276 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.810246 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c"] Jul 14 18:22:27.896668 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.896633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9fbh\" (UniqueName: \"kubernetes.io/projected/7ed78753-01af-4475-99c0-b9afda552c13-kube-api-access-q9fbh\") pod \"migrator-7f59b54f6c-s5g7c\" (UID: \"7ed78753-01af-4475-99c0-b9afda552c13\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" Jul 14 18:22:27.998020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:27.997942 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9fbh\" (UniqueName: \"kubernetes.io/projected/7ed78753-01af-4475-99c0-b9afda552c13-kube-api-access-q9fbh\") pod \"migrator-7f59b54f6c-s5g7c\" (UID: \"7ed78753-01af-4475-99c0-b9afda552c13\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" Jul 14 18:22:28.016266 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:28.016236 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9fbh\" (UniqueName: \"kubernetes.io/projected/7ed78753-01af-4475-99c0-b9afda552c13-kube-api-access-q9fbh\") pod \"migrator-7f59b54f6c-s5g7c\" (UID: \"7ed78753-01af-4475-99c0-b9afda552c13\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" Jul 14 18:22:28.108038 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:28.107954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" Jul 14 18:22:28.803985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:28.803951 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:28.804178 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:28.804014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:28.804178 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:28.804094 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:28.804178 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:28.804176 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:28.804343 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:28.804197 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.80417306 +0000 UTC m=+41.529313104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : configmap "kube-rbac-proxy" not found Jul 14 18:22:28.804343 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:28.804231 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.804214664 +0000 UTC m=+41.529354707 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:28.851665 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:28.851636 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6fzdg_c2b03911-b95b-46cb-bfca-20132d7d34a8/dns-node-resolver/0.log" Jul 14 18:22:29.304223 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:29.304045 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c"] Jul 14 18:22:29.307412 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:29.307387 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ed78753_01af_4475_99c0_b9afda552c13.slice/crio-076610a1d3f0beabb983922a01c544c31c25760e115ac646e13f29bcdd33a22b WatchSource:0}: Error finding container 076610a1d3f0beabb983922a01c544c31c25760e115ac646e13f29bcdd33a22b: Status 404 returned error can't find the container with id 076610a1d3f0beabb983922a01c544c31c25760e115ac646e13f29bcdd33a22b Jul 14 18:22:29.411091 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:29.411064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:29.411243 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:29.411224 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:29.411325 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:29.411312 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls podName:81ae0b10-8390-49bd-844c-18be5fb5a09c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:33.411286524 +0000 UTC m=+44.136426580 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls") pod "dns-default-stjst" (UID: "81ae0b10-8390-49bd-844c-18be5fb5a09c") : secret "dns-default-metrics-tls" not found Jul 14 18:22:30.041068 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.041036 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d05a5a-3286-4a73-825d-6b0c1fbd6325" containerID="71ee02545e02ed0562f4d3f04d6fc48dcb3719ebe8e22e9d6dadcf1c80cd15a8" exitCode=0 Jul 14 18:22:30.041255 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.041112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerDied","Data":"71ee02545e02ed0562f4d3f04d6fc48dcb3719ebe8e22e9d6dadcf1c80cd15a8"} Jul 14 18:22:30.042992 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.042964 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jzc5q" event={"ID":"13730500-ea99-40e3-a88a-07a5593ac288","Type":"ContainerStarted","Data":"f9e9dd3ffd37c377b61ef0f8fafdb3ecd0ad5ce5cae506f0b03c73a9b3593c06"} Jul 14 18:22:30.044397 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.044370 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" event={"ID":"7ed78753-01af-4475-99c0-b9afda552c13","Type":"ContainerStarted","Data":"076610a1d3f0beabb983922a01c544c31c25760e115ac646e13f29bcdd33a22b"} Jul 14 18:22:30.098076 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.097320 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jzc5q" podStartSLOduration=2.087260036 podStartE2EDuration="5.097299323s" totalImagesPullingTime="3.010039287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:30.095985977 +0000 UTC m=+40.821126038" watchObservedRunningTime="2026-07-14 18:22:30.097299323 +0000 UTC m=+40.822439385" Jul 14 18:22:30.197932 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.197908 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-98zcz"] Jul 14 18:22:30.219354 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.219330 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-98zcz"] Jul 14 18:22:30.219453 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.219420 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.225007 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.224980 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 14 18:22:30.225198 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.225034 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-j8955\"" Jul 14 18:22:30.225198 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.225090 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 14 18:22:30.225198 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.225131 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 14 18:22:30.225198 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.225171 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 14 18:22:30.252129 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.252108 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9k87_fcbafb7f-bcf0-485d-84f9-83783eef701f/node-ca/0.log" Jul 14 18:22:30.318023 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.317949 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/85f0c7fb-2e58-445d-9f59-1863a5962a50-signing-key\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.318475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.318021 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/85f0c7fb-2e58-445d-9f59-1863a5962a50-signing-cabundle\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.318475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.318049 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6nkz\" (UniqueName: \"kubernetes.io/projected/85f0c7fb-2e58-445d-9f59-1863a5962a50-kube-api-access-k6nkz\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.419244 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.419203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/85f0c7fb-2e58-445d-9f59-1863a5962a50-signing-cabundle\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.419392 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.419251 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6nkz\" (UniqueName: \"kubernetes.io/projected/85f0c7fb-2e58-445d-9f59-1863a5962a50-kube-api-access-k6nkz\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.419392 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.419340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/85f0c7fb-2e58-445d-9f59-1863a5962a50-signing-key\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.420000 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.419975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/85f0c7fb-2e58-445d-9f59-1863a5962a50-signing-cabundle\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.423611 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.423438 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/85f0c7fb-2e58-445d-9f59-1863a5962a50-signing-key\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.431071 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.431046 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6nkz\" (UniqueName: \"kubernetes.io/projected/85f0c7fb-2e58-445d-9f59-1863a5962a50-kube-api-access-k6nkz\") pod \"service-ca-79f68b9f4f-98zcz\" (UID: \"85f0c7fb-2e58-445d-9f59-1863a5962a50\") " pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.528092 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.528064 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" Jul 14 18:22:30.746310 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.746261 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-98zcz"] Jul 14 18:22:30.782706 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:30.782675 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f0c7fb_2e58_445d_9f59_1863a5962a50.slice/crio-eda2f9751585a6e25ea2f9f05125f8e4d590ec5d461416ee9c82cdd3ed037b3f WatchSource:0}: Error finding container eda2f9751585a6e25ea2f9f05125f8e4d590ec5d461416ee9c82cdd3ed037b3f: Status 404 returned error can't find the container with id eda2f9751585a6e25ea2f9f05125f8e4d590ec5d461416ee9c82cdd3ed037b3f Jul 14 18:22:30.822851 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.822830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:30.823033 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.822945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:30.823033 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:30.823004 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:30.823110 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:30.823060 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:30.823110 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:30.823072 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:34.823052025 +0000 UTC m=+45.548192082 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:30.823110 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:30.823096 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:34.823086218 +0000 UTC m=+45.548226283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : configmap "kube-rbac-proxy" not found Jul 14 18:22:30.857670 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:30.857650 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jzc5q_13730500-ea99-40e3-a88a-07a5593ac288/serve-healthcheck-canary/0.log" Jul 14 18:22:31.049456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:31.049425 2580 generic.go:356] "Generic (PLEG): container finished" podID="a2d05a5a-3286-4a73-825d-6b0c1fbd6325" containerID="37071a9c7c519dfbf255e70d9312050e14d0c86d72db0bd2d3ae2ce49c4a3436" exitCode=0 Jul 14 18:22:31.049585 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:31.049498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerDied","Data":"37071a9c7c519dfbf255e70d9312050e14d0c86d72db0bd2d3ae2ce49c4a3436"} Jul 14 18:22:31.051244 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:31.051221 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" event={"ID":"7ed78753-01af-4475-99c0-b9afda552c13","Type":"ContainerStarted","Data":"3f7360aadb9a68e9aec82d5566644304d2f0576c620184b4b80110392b7912db"} Jul 14 18:22:31.051328 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:31.051271 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" event={"ID":"7ed78753-01af-4475-99c0-b9afda552c13","Type":"ContainerStarted","Data":"c27b5792d7dbf5a5faa0771944756b475edb670c5143ba8f9165ccbd25bad76e"} Jul 14 18:22:31.052352 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:31.052330 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" event={"ID":"85f0c7fb-2e58-445d-9f59-1863a5962a50","Type":"ContainerStarted","Data":"eda2f9751585a6e25ea2f9f05125f8e4d590ec5d461416ee9c82cdd3ed037b3f"} Jul 14 18:22:31.102491 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:31.102444 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-s5g7c" podStartSLOduration=2.613548877 podStartE2EDuration="4.102429702s" totalImagesPullingTime="1.488880825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:31.102151899 +0000 UTC m=+41.827291960" watchObservedRunningTime="2026-07-14 18:22:31.102429702 +0000 UTC m=+41.827569763" Jul 14 18:22:32.059188 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:32.059150 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" event={"ID":"a2d05a5a-3286-4a73-825d-6b0c1fbd6325","Type":"ContainerStarted","Data":"715731b4cf8adf52eef9dff9e6ace4e6f23b8a245fe0f03f1a5bf31721165396"} Jul 14 18:22:33.443683 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:33.443579 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:33.444071 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:33.443753 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:33.444071 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:33.443863 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls podName:81ae0b10-8390-49bd-844c-18be5fb5a09c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:41.443840768 +0000 UTC m=+52.168980809 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls") pod "dns-default-stjst" (UID: "81ae0b10-8390-49bd-844c-18be5fb5a09c") : secret "dns-default-metrics-tls" not found Jul 14 18:22:34.066169 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:34.066117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" event={"ID":"85f0c7fb-2e58-445d-9f59-1863a5962a50","Type":"ContainerStarted","Data":"260b5daab4eac23143bc6802f965c0b36045cafd586a12c34f2825a62a46bd31"} Jul 14 18:22:34.102673 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:34.102434 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k8lfb" podStartSLOduration=11.059264083 podStartE2EDuration="45.102416014s" totalImagesPullingTime="34.043151931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:32.096707121 +0000 UTC m=+42.821847182" watchObservedRunningTime="2026-07-14 18:22:34.102416014 +0000 UTC m=+44.827556074" Jul 14 18:22:34.103314 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:34.103278 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-98zcz" podStartSLOduration=1.790365384 podStartE2EDuration="4.103267826s" totalImagesPullingTime="2.312902442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:34.101787442 +0000 UTC m=+44.826927504" watchObservedRunningTime="2026-07-14 18:22:34.103267826 +0000 UTC m=+44.828407887" Jul 14 18:22:34.854878 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:34.854781 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:34.854878 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:34.854860 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:34.855441 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:34.854974 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:34.855441 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:22:34.855034 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls podName:85330e72-de6f-4c21-a6b8-a912f561c69d nodeName:}" failed. No retries permitted until 2026-07-14 18:22:42.855015751 +0000 UTC m=+53.580155805 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2qs8b" (UID: "85330e72-de6f-4c21-a6b8-a912f561c69d") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:34.855624 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:34.855602 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85330e72-de6f-4c21-a6b8-a912f561c69d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:41.504609 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:41.504563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:41.507566 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:41.507542 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/81ae0b10-8390-49bd-844c-18be5fb5a09c-metrics-tls\") pod \"dns-default-stjst\" (UID: \"81ae0b10-8390-49bd-844c-18be5fb5a09c\") " pod="openshift-dns/dns-default-stjst" Jul 14 18:22:41.595994 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:41.595956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-stjst" Jul 14 18:22:41.754178 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:41.754141 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-stjst"] Jul 14 18:22:41.757669 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:41.757635 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ae0b10_8390_49bd_844c_18be5fb5a09c.slice/crio-00fa5cc182874d24433f2182bd010c61aeaab699741393a3de5ee807e3eba65e WatchSource:0}: Error finding container 00fa5cc182874d24433f2182bd010c61aeaab699741393a3de5ee807e3eba65e: Status 404 returned error can't find the container with id 00fa5cc182874d24433f2182bd010c61aeaab699741393a3de5ee807e3eba65e Jul 14 18:22:42.087604 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:42.087564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-stjst" event={"ID":"81ae0b10-8390-49bd-844c-18be5fb5a09c","Type":"ContainerStarted","Data":"00fa5cc182874d24433f2182bd010c61aeaab699741393a3de5ee807e3eba65e"} Jul 14 18:22:42.916087 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:42.916048 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:42.918857 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:42.918832 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85330e72-de6f-4c21-a6b8-a912f561c69d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2qs8b\" (UID: \"85330e72-de6f-4c21-a6b8-a912f561c69d\") " pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:42.991214 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:42.990923 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2qs8b" Jul 14 18:22:43.184185 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:43.184103 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2qs8b"] Jul 14 18:22:43.190373 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:43.190327 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85330e72_de6f_4c21_a6b8_a912f561c69d.slice/crio-d4d7e9100e571ebd9ac0202bd13c78b62e4170ee70a44ebded86d6f43676b924 WatchSource:0}: Error finding container d4d7e9100e571ebd9ac0202bd13c78b62e4170ee70a44ebded86d6f43676b924: Status 404 returned error can't find the container with id d4d7e9100e571ebd9ac0202bd13c78b62e4170ee70a44ebded86d6f43676b924 Jul 14 18:22:44.094774 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:44.094715 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qs8b" event={"ID":"85330e72-de6f-4c21-a6b8-a912f561c69d","Type":"ContainerStarted","Data":"d4d7e9100e571ebd9ac0202bd13c78b62e4170ee70a44ebded86d6f43676b924"} Jul 14 18:22:45.101993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:45.101958 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-stjst" event={"ID":"81ae0b10-8390-49bd-844c-18be5fb5a09c","Type":"ContainerStarted","Data":"2d4be61ca4223f63891e4ee8fd28b0ee70c3d292e876c2165727a6870790e6e9"} Jul 14 18:22:45.102377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:45.102017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-stjst" event={"ID":"81ae0b10-8390-49bd-844c-18be5fb5a09c","Type":"ContainerStarted","Data":"8cbd5be477f9f4bc19d7edf0f06f648a09b145ff8ac38496bdaa8bbc3d633cd1"} Jul 14 18:22:45.102377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:45.102059 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-stjst" Jul 14 18:22:50.023006 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:50.022975 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9hs2p" Jul 14 18:22:50.060510 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:50.060133 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-stjst" podStartSLOduration=22.682651214 podStartE2EDuration="25.060112294s" totalImagesPullingTime="2.37746108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:45.131388498 +0000 UTC m=+55.856528558" watchObservedRunningTime="2026-07-14 18:22:50.060112294 +0000 UTC m=+60.785252352" Jul 14 18:22:51.124399 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:51.124365 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qs8b" event={"ID":"85330e72-de6f-4c21-a6b8-a912f561c69d","Type":"ContainerStarted","Data":"c8637f0f6d22bc02452319555f0b40d9ceb8f28bf69276440a4420b6e696a23c"} Jul 14 18:22:52.129626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:52.129592 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qs8b" event={"ID":"85330e72-de6f-4c21-a6b8-a912f561c69d","Type":"ContainerStarted","Data":"a2d5d601dbdc86476a0a2edb39cd5aa71fc5d9eddda40d80c8a748cc33012ba6"} Jul 14 18:22:54.346473 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.346396 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5"] Jul 14 18:22:54.349475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.349454 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.352013 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.351995 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 14 18:22:54.353012 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.352970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 18:22:54.353101 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.353020 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 14 18:22:54.353101 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.353036 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 14 18:22:54.353101 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.353059 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 14 18:22:54.353313 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.353296 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 18:22:54.353401 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.353385 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 18:22:54.404260 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404230 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5"] Jul 14 18:22:54.404452 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404434 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-ca\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.404495 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvr89\" (UniqueName: \"kubernetes.io/projected/10ab33c6-a354-4b44-aadf-438f68942b5d-kube-api-access-nvr89\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.404495 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-hub\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.404573 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab33c6-a354-4b44-aadf-438f68942b5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.404573 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.404682 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.404666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.505700 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.505670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvr89\" (UniqueName: \"kubernetes.io/projected/10ab33c6-a354-4b44-aadf-438f68942b5d-kube-api-access-nvr89\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.505700 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.505700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-hub\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.505959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.505719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab33c6-a354-4b44-aadf-438f68942b5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.505959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.505736 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.505959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.505833 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.505959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.505901 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-ca\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.506529 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.506479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/10ab33c6-a354-4b44-aadf-438f68942b5d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.508365 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.508335 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-ca\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.508473 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.508449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.508696 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.508681 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-hub\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.508821 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.508804 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/10ab33c6-a354-4b44-aadf-438f68942b5d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.547242 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.547213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvr89\" (UniqueName: \"kubernetes.io/projected/10ab33c6-a354-4b44-aadf-438f68942b5d-kube-api-access-nvr89\") pod \"cluster-proxy-proxy-agent-786c5bc649-gchf5\" (UID: \"10ab33c6-a354-4b44-aadf-438f68942b5d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.676650 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.676579 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" Jul 14 18:22:54.859400 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:54.859344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5"] Jul 14 18:22:54.862115 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:54.862088 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod10ab33c6_a354_4b44_aadf_438f68942b5d.slice/crio-806a3a5992856d2ccaef95aa5d37683421f88d2df73933099268073f96a63866 WatchSource:0}: Error finding container 806a3a5992856d2ccaef95aa5d37683421f88d2df73933099268073f96a63866: Status 404 returned error can't find the container with id 806a3a5992856d2ccaef95aa5d37683421f88d2df73933099268073f96a63866 Jul 14 18:22:55.107731 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.107707 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-stjst" Jul 14 18:22:55.140898 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.140862 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2qs8b" event={"ID":"85330e72-de6f-4c21-a6b8-a912f561c69d","Type":"ContainerStarted","Data":"c203b9730ced59b9722a38762574d9d8effb45537a24baf4a927b5434dcd0c72"} Jul 14 18:22:55.142063 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.142032 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" event={"ID":"10ab33c6-a354-4b44-aadf-438f68942b5d","Type":"ContainerStarted","Data":"806a3a5992856d2ccaef95aa5d37683421f88d2df73933099268073f96a63866"} Jul 14 18:22:55.199785 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.199704 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2qs8b" podStartSLOduration=17.477298716 podStartE2EDuration="28.199680669s" totalImagesPullingTime="10.722381953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:22:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:55.198736885 +0000 UTC m=+65.923876947" watchObservedRunningTime="2026-07-14 18:22:55.199680669 +0000 UTC m=+65.924820731" Jul 14 18:22:55.413208 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.413099 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:55.416228 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.416058 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:22:55.426307 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.426268 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f55ab0b2-0a83-4a87-9649-f7e71ffea294-original-pull-secret\") pod \"global-pull-secret-syncer-65qbb\" (UID: \"f55ab0b2-0a83-4a87-9649-f7e71ffea294\") " pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:55.501810 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.501779 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-65qbb" Jul 14 18:22:55.514748 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.514373 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:55.516939 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.516899 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:22:55.528509 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.528487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fc9a1bc7-930e-473c-b344-01a70ba50626-metrics-certs\") pod \"network-metrics-daemon-5wktq\" (UID: \"fc9a1bc7-930e-473c-b344-01a70ba50626\") " pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:55.615460 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.615424 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:55.618544 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.618522 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:22:55.628673 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.628477 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:22:55.639562 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.639534 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjfqr\" (UniqueName: \"kubernetes.io/projected/845e9932-2592-4385-b8a2-229a236a8700-kube-api-access-gjfqr\") pod \"network-check-target-wc7l6\" (UID: \"845e9932-2592-4385-b8a2-229a236a8700\") " pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:55.658227 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.658149 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-65qbb"] Jul 14 18:22:55.661433 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:55.661400 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf55ab0b2_0a83_4a87_9649_f7e71ffea294.slice/crio-950c02348f3929f1bd3aff08fb8b7f2c47710cdf8850d7c7a051d35553618858 WatchSource:0}: Error finding container 950c02348f3929f1bd3aff08fb8b7f2c47710cdf8850d7c7a051d35553618858: Status 404 returned error can't find the container with id 950c02348f3929f1bd3aff08fb8b7f2c47710cdf8850d7c7a051d35553618858 Jul 14 18:22:55.795177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.795144 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lzfqw\"" Jul 14 18:22:55.799671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.799644 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:22:55.799825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.799705 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzvpc\"" Jul 14 18:22:55.808219 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.808040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5wktq" Jul 14 18:22:55.967562 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.967464 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wc7l6"] Jul 14 18:22:55.973030 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:55.972972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845e9932_2592_4385_b8a2_229a236a8700.slice/crio-9ec9d49e536c6c60c7d018231305ccc48d22a11344608375d98890717f5f2827 WatchSource:0}: Error finding container 9ec9d49e536c6c60c7d018231305ccc48d22a11344608375d98890717f5f2827: Status 404 returned error can't find the container with id 9ec9d49e536c6c60c7d018231305ccc48d22a11344608375d98890717f5f2827 Jul 14 18:22:55.987244 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:55.987216 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5wktq"] Jul 14 18:22:55.991253 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:22:55.991225 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc9a1bc7_930e_473c_b344_01a70ba50626.slice/crio-a960534eb3b36c4cbbe566f461698bb576453c3229505990567877004c5e24a4 WatchSource:0}: Error finding container a960534eb3b36c4cbbe566f461698bb576453c3229505990567877004c5e24a4: Status 404 returned error can't find the container with id a960534eb3b36c4cbbe566f461698bb576453c3229505990567877004c5e24a4 Jul 14 18:22:56.150311 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:56.150275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wc7l6" event={"ID":"845e9932-2592-4385-b8a2-229a236a8700","Type":"ContainerStarted","Data":"9ec9d49e536c6c60c7d018231305ccc48d22a11344608375d98890717f5f2827"} Jul 14 18:22:56.152049 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:56.151990 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65qbb" event={"ID":"f55ab0b2-0a83-4a87-9649-f7e71ffea294","Type":"ContainerStarted","Data":"950c02348f3929f1bd3aff08fb8b7f2c47710cdf8850d7c7a051d35553618858"} Jul 14 18:22:56.153858 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:22:56.153754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wktq" event={"ID":"fc9a1bc7-930e-473c-b344-01a70ba50626","Type":"ContainerStarted","Data":"a960534eb3b36c4cbbe566f461698bb576453c3229505990567877004c5e24a4"} Jul 14 18:23:03.191175 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.191135 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" event={"ID":"10ab33c6-a354-4b44-aadf-438f68942b5d","Type":"ContainerStarted","Data":"b1aa4892c019ea60d7f1d2aa13b2aa97dd9a725e8a28a74d9588ebdc44126151"} Jul 14 18:23:03.191627 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.191187 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" event={"ID":"10ab33c6-a354-4b44-aadf-438f68942b5d","Type":"ContainerStarted","Data":"c9f25019ad92b1d4f2d5c1703e796ff78cb157b31e139e2fc6ba6fb501b1ce26"} Jul 14 18:23:03.191627 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.191201 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" event={"ID":"10ab33c6-a354-4b44-aadf-438f68942b5d","Type":"ContainerStarted","Data":"5c0a81cc5116208a9b7bcdbcdfe191761a5a274682cba11f1f39d86820d7a17d"} Jul 14 18:23:03.192960 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.192938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wktq" event={"ID":"fc9a1bc7-930e-473c-b344-01a70ba50626","Type":"ContainerStarted","Data":"39bc75d41cc1db90ccd41f72da94d65ef0fda5afa79a6d6a843861ad008f4a57"} Jul 14 18:23:03.193059 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.192980 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5wktq" event={"ID":"fc9a1bc7-930e-473c-b344-01a70ba50626","Type":"ContainerStarted","Data":"bb56bfe2980b22bdb9be0619f2ce209e8737f26773e7caa82aadea9e23625209"} Jul 14 18:23:03.194365 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.194346 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wc7l6" event={"ID":"845e9932-2592-4385-b8a2-229a236a8700","Type":"ContainerStarted","Data":"535acc889d12ee1581b67d39f5786393c2f11165ac6bc258e3312e5e4286338f"} Jul 14 18:23:03.194432 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.194390 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:23:03.195698 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.195684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-65qbb" event={"ID":"f55ab0b2-0a83-4a87-9649-f7e71ffea294","Type":"ContainerStarted","Data":"d887015d3e278afa25fe494c1ba4bda95c690ac09a4f81fdcb595808d8a7be63"} Jul 14 18:23:03.219727 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.219685 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-786c5bc649-gchf5" podStartSLOduration=1.9509768090000001 podStartE2EDuration="9.2196733s" totalImagesPullingTime="7.268696491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:03.218055937 +0000 UTC m=+73.943195999" watchObservedRunningTime="2026-07-14 18:23:03.2196733 +0000 UTC m=+73.944813361" Jul 14 18:23:03.243372 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.243330 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wc7l6" podStartSLOduration=68.081041747 podStartE2EDuration="1m14.243314487s" totalImagesPullingTime="6.16227274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:03.242903284 +0000 UTC m=+73.968043349" watchObservedRunningTime="2026-07-14 18:23:03.243314487 +0000 UTC m=+73.968454549" Jul 14 18:23:03.259680 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.259638 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-65qbb" podStartSLOduration=67.777701843 podStartE2EDuration="1m14.259630046s" totalImagesPullingTime="6.481928203s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:03.258872574 +0000 UTC m=+73.984012645" watchObservedRunningTime="2026-07-14 18:23:03.259630046 +0000 UTC m=+73.984770106" Jul 14 18:23:03.275771 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:03.275718 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5wktq" podStartSLOduration=68.135804188 podStartE2EDuration="1m14.275706428s" totalImagesPullingTime="6.13990224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:03.274816094 +0000 UTC m=+73.999956155" watchObservedRunningTime="2026-07-14 18:23:03.275706428 +0000 UTC m=+74.000846489" Jul 14 18:23:06.012246 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.012210 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nf7wx"] Jul 14 18:23:06.015541 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.015518 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.020735 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.020717 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 18:23:06.021003 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.020988 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2g88\"" Jul 14 18:23:06.022135 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.022111 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 18:23:06.022357 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.022237 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 18:23:06.022357 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.022238 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 18:23:06.022357 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.022314 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 18:23:06.022357 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.022182 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 18:23:06.100312 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100312 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-root\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100335 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/960c371c-0d49-4558-85cb-3347d8b691d8-metrics-client-ca\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100428 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-wtmp\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100488 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-tls\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-textfile\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100748 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r6zh\" (UniqueName: \"kubernetes.io/projected/960c371c-0d49-4558-85cb-3347d8b691d8-kube-api-access-8r6zh\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100748 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100654 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.100748 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.100703 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-sys\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201228 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201197 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-textfile\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r6zh\" (UniqueName: \"kubernetes.io/projected/960c371c-0d49-4558-85cb-3347d8b691d8-kube-api-access-8r6zh\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201254 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201288 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-sys\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201330 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-root\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201351 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/960c371c-0d49-4558-85cb-3347d8b691d8-metrics-client-ca\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201371 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-wtmp\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201393 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-tls\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201836 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201442 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-root\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201836 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-sys\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201836 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-wtmp\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.201836 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.201586 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-textfile\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.202107 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.202086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/960c371c-0d49-4558-85cb-3347d8b691d8-metrics-client-ca\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.202163 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.202086 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-accelerators-collector-config\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.204148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.204122 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.204288 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.204269 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/960c371c-0d49-4558-85cb-3347d8b691d8-node-exporter-tls\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.210019 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.209995 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r6zh\" (UniqueName: \"kubernetes.io/projected/960c371c-0d49-4558-85cb-3347d8b691d8-kube-api-access-8r6zh\") pod \"node-exporter-nf7wx\" (UID: \"960c371c-0d49-4558-85cb-3347d8b691d8\") " pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.326293 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:06.326210 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nf7wx" Jul 14 18:23:06.340384 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:23:06.340353 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod960c371c_0d49_4558_85cb_3347d8b691d8.slice/crio-2d90bae6148d0cbf0bb081d38562115599c957a8e25a58225ead3a91d87cc779 WatchSource:0}: Error finding container 2d90bae6148d0cbf0bb081d38562115599c957a8e25a58225ead3a91d87cc779: Status 404 returned error can't find the container with id 2d90bae6148d0cbf0bb081d38562115599c957a8e25a58225ead3a91d87cc779 Jul 14 18:23:07.210194 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:07.210156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nf7wx" event={"ID":"960c371c-0d49-4558-85cb-3347d8b691d8","Type":"ContainerStarted","Data":"2d90bae6148d0cbf0bb081d38562115599c957a8e25a58225ead3a91d87cc779"} Jul 14 18:23:08.214895 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:08.214854 2580 generic.go:356] "Generic (PLEG): container finished" podID="960c371c-0d49-4558-85cb-3347d8b691d8" containerID="923d45b98e9e5ee0330314b413e0a8ae8f73afc6962def5b9dcd2c4e94001a3b" exitCode=0 Jul 14 18:23:08.215326 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:08.214924 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nf7wx" event={"ID":"960c371c-0d49-4558-85cb-3347d8b691d8","Type":"ContainerDied","Data":"923d45b98e9e5ee0330314b413e0a8ae8f73afc6962def5b9dcd2c4e94001a3b"} Jul 14 18:23:09.220689 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:09.220650 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nf7wx" event={"ID":"960c371c-0d49-4558-85cb-3347d8b691d8","Type":"ContainerStarted","Data":"56f18f5c0f89e49db403ace59ad110b5ee467f54e12216dcacd243c057d1f3c1"} Jul 14 18:23:09.221155 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:09.220745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nf7wx" event={"ID":"960c371c-0d49-4558-85cb-3347d8b691d8","Type":"ContainerStarted","Data":"6f9c6bb4d47607f847b0daa5ed4f65dd7fb54d7dfc76178253ac19977b9a0743"} Jul 14 18:23:12.505481 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.505352 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nf7wx" podStartSLOduration=6.524869253 podStartE2EDuration="7.505333834s" totalImagesPullingTime="906.015078ms" totalInitContainerRuntime="74.449503ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:09.246258098 +0000 UTC m=+79.971398169" watchObservedRunningTime="2026-07-14 18:23:12.505333834 +0000 UTC m=+83.230473895" Jul 14 18:23:12.506187 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.506170 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:23:12.510373 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.510358 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.517199 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.517179 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 14 18:23:12.517539 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.517181 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 14 18:23:12.517690 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.517669 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 18:23:12.517780 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.517711 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 14 18:23:12.517974 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.517952 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 14 18:23:12.518129 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.518112 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 14 18:23:12.518359 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.518340 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 14 18:23:12.520055 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.519857 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 14 18:23:12.520745 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.520559 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 14 18:23:12.520745 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.520729 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 14 18:23:12.521292 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.521270 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-499r344tco89s\"" Jul 14 18:23:12.521449 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.521427 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vfqt2\"" Jul 14 18:23:12.522047 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.522028 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 14 18:23:12.522873 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.522854 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 14 18:23:12.522946 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.522854 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 14 18:23:12.523963 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.523948 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 14 18:23:12.525998 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.525981 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 14 18:23:12.535245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.535226 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:23:12.648220 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648190 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648220 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648225 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648278 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648296 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648372 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648400 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648479 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config-out\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648504 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648531 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648550 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648570 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-web-config\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648599 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbxk4\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-kube-api-access-mbxk4\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648646 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648675 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.648720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.648697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.749867 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749822 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.749867 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749868 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749907 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config-out\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749929 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.749999 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-web-config\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.750539 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.750506 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.750867 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbxk4\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-kube-api-access-mbxk4\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.750911 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.750959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.750998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751084 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751093 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751158 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.751456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751261 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.752207 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.751563 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.752526 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.752503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.752705 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.752679 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.753454 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.753423 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.753576 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.753550 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.754240 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.754213 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-web-config\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.754240 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.754232 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.754390 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.754313 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.755313 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.755270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.755709 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.755643 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config-out\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.756100 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.756076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.756253 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.756226 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.756467 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.756445 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.756850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.756827 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.757001 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.756982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.757153 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.757138 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.775727 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.775705 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbxk4\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-kube-api-access-mbxk4\") pod \"prometheus-k8s-0\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.820898 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.820865 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:12.967419 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:12.967313 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:23:12.969882 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:23:12.969856 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2958bab5_23e1_4b6c_ba1a_bf3d0bc06225.slice/crio-17e36516a542c22ab05c16ebe2b41cb14f3dcbb8eb26a2fc7dd79443a88c3a33 WatchSource:0}: Error finding container 17e36516a542c22ab05c16ebe2b41cb14f3dcbb8eb26a2fc7dd79443a88c3a33: Status 404 returned error can't find the container with id 17e36516a542c22ab05c16ebe2b41cb14f3dcbb8eb26a2fc7dd79443a88c3a33 Jul 14 18:23:13.233880 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:13.233844 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"17e36516a542c22ab05c16ebe2b41cb14f3dcbb8eb26a2fc7dd79443a88c3a33"} Jul 14 18:23:14.239601 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:14.239563 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" exitCode=0 Jul 14 18:23:14.239998 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:14.239607 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} Jul 14 18:23:21.269699 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:21.269657 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} Jul 14 18:23:21.270101 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:21.269749 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} Jul 14 18:23:23.280673 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:23.280636 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} Jul 14 18:23:23.281126 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:23.280700 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} Jul 14 18:23:23.281126 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:23.280725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} Jul 14 18:23:23.281126 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:23.280750 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerStarted","Data":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} Jul 14 18:23:23.343008 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:23.342950 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.215003429 podStartE2EDuration="11.342931256s" totalImagesPullingTime="9.110785673s" totalInitContainerRuntime="17.142154ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:12 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:23.336627098 +0000 UTC m=+94.061767161" watchObservedRunningTime="2026-07-14 18:23:23.342931256 +0000 UTC m=+94.068071317" Jul 14 18:23:27.820999 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:27.820967 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:23:34.201377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:23:34.201347 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wc7l6" Jul 14 18:24:12.821657 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:12.821624 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:12.840706 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:12.840682 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:13.467582 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:13.467556 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:30.878047 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878011 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:24:30.878475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878291 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="prometheus" containerID="cri-o://588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" gracePeriod=600 Jul 14 18:24:30.878475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878305 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy" containerID="cri-o://4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" gracePeriod=600 Jul 14 18:24:30.878475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878313 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="thanos-sidecar" containerID="cri-o://552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" gracePeriod=600 Jul 14 18:24:30.878475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878380 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-thanos" containerID="cri-o://759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" gracePeriod=600 Jul 14 18:24:30.878475 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878411 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="config-reloader" containerID="cri-o://f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" gracePeriod=600 Jul 14 18:24:30.878736 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:30.878555 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-web" containerID="cri-o://5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" gracePeriod=600 Jul 14 18:24:31.116539 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.116515 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.243705 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243617 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-1\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243705 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243660 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-db\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243705 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243692 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-2\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243719 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-kubelet-serving-ca-bundle\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243743 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-grpc-tls\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243794 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243837 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbxk4\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-kube-api-access-mbxk4\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243872 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-metrics-client-ca\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243901 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-kube-rbac-proxy\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243958 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-tls-assets\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.243993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.243986 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-thanos-prometheus-http-client-file\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244018 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244048 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244082 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config-out\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244091 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244125 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-trusted-ca-bundle\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244149 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-web-config\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244179 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-tls\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244181 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244203 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-serving-certs-ca-bundle\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244230 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-0\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244268 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-metrics-client-certs\") pod \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\" (UID: \"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225\") " Jul 14 18:24:31.244363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244317 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.245176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244502 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.245176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244521 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.245176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244538 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.245176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.244982 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.245376 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.245322 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:24:31.245904 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.245460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.246001 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.245898 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.247083 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.247054 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.247176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.247102 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.247292 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.247258 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config-out" (OuterVolumeSpecName: "config-out") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:24:31.247636 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.247608 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:24:31.249112 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config" (OuterVolumeSpecName: "config") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.249333 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249258 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.249333 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249296 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.249461 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249337 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.249461 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249349 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.249461 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249444 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.249570 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249545 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:24:31.249640 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.249625 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-kube-api-access-mbxk4" (OuterVolumeSpecName: "kube-api-access-mbxk4") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "kube-api-access-mbxk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:24:31.257650 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.257630 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-web-config" (OuterVolumeSpecName: "web-config") pod "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" (UID: "2958bab5-23e1-4b6c-ba1a-bf3d0bc06225"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:24:31.345492 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345465 2580 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-thanos-prometheus-http-client-file\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345492 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345488 2580 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345497 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345507 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-config-out\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345516 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-trusted-ca-bundle\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345524 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-web-config\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345532 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-tls\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345541 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345551 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345559 2580 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-metrics-client-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345567 2580 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-prometheus-k8s-db\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345575 2580 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-grpc-tls\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345583 2580 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345591 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbxk4\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-kube-api-access-mbxk4\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345599 2580 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-configmap-metrics-client-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345607 2580 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-secret-kube-rbac-proxy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.345638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.345616 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225-tls-assets\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:24:31.516636 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516608 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" exitCode=0 Jul 14 18:24:31.516636 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516631 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" exitCode=0 Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516644 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" exitCode=0 Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516652 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" exitCode=0 Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516657 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" exitCode=0 Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516662 2580 generic.go:356] "Generic (PLEG): container finished" podID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" exitCode=0 Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516695 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} Jul 14 18:24:31.516850 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} Jul 14 18:24:31.517052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} Jul 14 18:24:31.517052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516884 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.517052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516899 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} Jul 14 18:24:31.517052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} Jul 14 18:24:31.517052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.516979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} Jul 14 18:24:31.517052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.517019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2958bab5-23e1-4b6c-ba1a-bf3d0bc06225","Type":"ContainerDied","Data":"17e36516a542c22ab05c16ebe2b41cb14f3dcbb8eb26a2fc7dd79443a88c3a33"} Jul 14 18:24:31.527384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.527365 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.533734 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.533716 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.539960 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.539941 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.546923 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.546899 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:24:31.547218 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.547196 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.558503 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.556209 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:24:31.558693 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.558670 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.566077 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.566056 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.572626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.572606 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.572919 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.572898 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.572975 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.572927 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} err="failed to get container status \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" Jul 14 18:24:31.572975 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.572963 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.573188 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.573170 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.573236 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573192 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} err="failed to get container status \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" Jul 14 18:24:31.573236 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573208 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.573402 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.573383 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.573440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573410 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} err="failed to get container status \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" Jul 14 18:24:31.573440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573425 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.573618 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.573600 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.573657 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573623 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} err="failed to get container status \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" Jul 14 18:24:31.573657 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573636 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.573839 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.573816 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.573908 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573841 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} err="failed to get container status \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" Jul 14 18:24:31.573908 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.573852 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.574097 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.574080 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.574135 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574101 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} err="failed to get container status \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" Jul 14 18:24:31.574135 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574113 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.574339 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:24:31.574324 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.574377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574342 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} err="failed to get container status \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" Jul 14 18:24:31.574377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574356 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.574569 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574550 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} err="failed to get container status \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" Jul 14 18:24:31.574645 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574572 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.574787 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574751 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} err="failed to get container status \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" Jul 14 18:24:31.574830 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574788 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.574961 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574945 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} err="failed to get container status \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" Jul 14 18:24:31.575002 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.574961 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.575120 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575106 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} err="failed to get container status \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" Jul 14 18:24:31.575162 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575120 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.575311 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575295 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} err="failed to get container status \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" Jul 14 18:24:31.575360 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575311 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.575505 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575488 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} err="failed to get container status \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" Jul 14 18:24:31.575571 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575507 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.575747 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575729 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} err="failed to get container status \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" Jul 14 18:24:31.575822 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575748 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.575985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575967 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} err="failed to get container status \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" Jul 14 18:24:31.576067 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.575986 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.576197 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576181 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} err="failed to get container status \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" Jul 14 18:24:31.576236 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576198 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.576380 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576362 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} err="failed to get container status \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" Jul 14 18:24:31.576427 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576380 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.576572 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576555 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} err="failed to get container status \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" Jul 14 18:24:31.576644 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576573 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.576788 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576743 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} err="failed to get container status \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" Jul 14 18:24:31.576788 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576772 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.576942 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576927 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} err="failed to get container status \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" Jul 14 18:24:31.576977 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.576942 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.577121 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577103 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} err="failed to get container status \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" Jul 14 18:24:31.577121 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577118 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.577336 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577317 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} err="failed to get container status \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" Jul 14 18:24:31.577380 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577337 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.577521 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577505 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} err="failed to get container status \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" Jul 14 18:24:31.577576 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577522 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.577703 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577688 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} err="failed to get container status \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" Jul 14 18:24:31.577703 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577703 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.577930 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577913 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} err="failed to get container status \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" Jul 14 18:24:31.577930 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.577930 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.578162 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578146 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} err="failed to get container status \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" Jul 14 18:24:31.578205 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578162 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.578369 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578354 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} err="failed to get container status \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" Jul 14 18:24:31.578410 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578369 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.578584 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578568 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} err="failed to get container status \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" Jul 14 18:24:31.578625 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578584 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.578801 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578783 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} err="failed to get container status \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" Jul 14 18:24:31.578892 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578801 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.578990 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578974 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} err="failed to get container status \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" Jul 14 18:24:31.578990 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.578989 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.579183 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579164 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} err="failed to get container status \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" Jul 14 18:24:31.579222 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579185 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.579359 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579342 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} err="failed to get container status \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" Jul 14 18:24:31.579359 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579358 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.579533 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579515 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} err="failed to get container status \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" Jul 14 18:24:31.579595 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579533 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.579735 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579722 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} err="failed to get container status \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" Jul 14 18:24:31.579802 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579736 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.579940 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579923 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} err="failed to get container status \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" Jul 14 18:24:31.579992 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.579940 2580 scope.go:117] "RemoveContainer" containerID="759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc" Jul 14 18:24:31.580141 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580126 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc"} err="failed to get container status \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": rpc error: code = NotFound desc = could not find container \"759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc\": container with ID starting with 759f15f4f5ec76b36f902bbcc5f3ea7f0701894e01ae4aaa086fc29bbafe6fcc not found: ID does not exist" Jul 14 18:24:31.580141 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580141 2580 scope.go:117] "RemoveContainer" containerID="4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc" Jul 14 18:24:31.580342 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580327 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc"} err="failed to get container status \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": rpc error: code = NotFound desc = could not find container \"4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc\": container with ID starting with 4054f621b8219eebc470ae35574d941b2cb438e33cd25f24b94ab2f5c8ea32dc not found: ID does not exist" Jul 14 18:24:31.580382 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580342 2580 scope.go:117] "RemoveContainer" containerID="5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759" Jul 14 18:24:31.580520 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580506 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759"} err="failed to get container status \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": rpc error: code = NotFound desc = could not find container \"5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759\": container with ID starting with 5df3d7c556b2abed87f5365f1ffa31fbdfcc64062880454220be3af369a49759 not found: ID does not exist" Jul 14 18:24:31.580560 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580520 2580 scope.go:117] "RemoveContainer" containerID="552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54" Jul 14 18:24:31.580685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580669 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54"} err="failed to get container status \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": rpc error: code = NotFound desc = could not find container \"552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54\": container with ID starting with 552c9322897a9e74922fe4919d070b3c82eae3dcb65afb6248dd866d5a97dd54 not found: ID does not exist" Jul 14 18:24:31.580724 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580685 2580 scope.go:117] "RemoveContainer" containerID="f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e" Jul 14 18:24:31.580864 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580846 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e"} err="failed to get container status \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": rpc error: code = NotFound desc = could not find container \"f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e\": container with ID starting with f39b9f31bb155532a38e34601b3c83d7e7baa2a24651643bbd96f2ccfcc6582e not found: ID does not exist" Jul 14 18:24:31.580913 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.580864 2580 scope.go:117] "RemoveContainer" containerID="588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689" Jul 14 18:24:31.581034 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.581018 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689"} err="failed to get container status \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": rpc error: code = NotFound desc = could not find container \"588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689\": container with ID starting with 588ec60c543a8aa7e79f9c8193cd73d227e46a4e2cdd182845bcdf6519049689 not found: ID does not exist" Jul 14 18:24:31.581070 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.581035 2580 scope.go:117] "RemoveContainer" containerID="305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57" Jul 14 18:24:31.581208 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.581193 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57"} err="failed to get container status \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": rpc error: code = NotFound desc = could not find container \"305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57\": container with ID starting with 305d04df8f2780101ce8fd1f737bd5f280cb83bc1c0b40261dfa31a055b15b57 not found: ID does not exist" Jul 14 18:24:31.594952 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.594929 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:24:31.595186 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595174 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="init-config-reloader" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595187 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="init-config-reloader" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595195 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="prometheus" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595200 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="prometheus" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595211 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="config-reloader" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595217 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="config-reloader" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595223 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-web" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595228 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-web" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595235 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="thanos-sidecar" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595240 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="thanos-sidecar" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595246 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy" Jul 14 18:24:31.595245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595251 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595257 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-thanos" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595262 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-thanos" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595296 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="thanos-sidecar" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595304 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="prometheus" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595311 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="config-reloader" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595316 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-web" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595321 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy" Jul 14 18:24:31.595587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.595326 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" containerName="kube-rbac-proxy-thanos" Jul 14 18:24:31.600343 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.600325 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.603919 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.603900 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 14 18:24:31.604016 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.603962 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 14 18:24:31.604204 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.604187 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 14 18:24:31.604269 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.604252 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 14 18:24:31.604307 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.604252 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 14 18:24:31.604616 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.604601 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 14 18:24:31.605368 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.605352 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 18:24:31.605684 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.605666 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 14 18:24:31.605684 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.605683 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 14 18:24:31.605858 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.605829 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-499r344tco89s\"" Jul 14 18:24:31.606247 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.606233 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 14 18:24:31.607271 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.607203 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 14 18:24:31.607435 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.607419 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 14 18:24:31.607654 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.607637 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-vfqt2\"" Jul 14 18:24:31.608712 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.608695 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 14 18:24:31.611179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.611164 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 14 18:24:31.614302 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.614287 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 14 18:24:31.643035 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.643006 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:24:31.748587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748559 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748587 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748615 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39fa3c3a-e65b-47e3-85f4-f653277da65c-config-out\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748699 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39fa3c3a-e65b-47e3-85f4-f653277da65c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-config\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.748825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748827 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.748978 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749029 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-web-config\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749426 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.749426 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.749185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llmrp\" (UniqueName: \"kubernetes.io/projected/39fa3c3a-e65b-47e3-85f4-f653277da65c-kube-api-access-llmrp\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.781998 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.781931 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2958bab5-23e1-4b6c-ba1a-bf3d0bc06225" path="/var/lib/kubelet/pods/2958bab5-23e1-4b6c-ba1a-bf3d0bc06225/volumes" Jul 14 18:24:31.849705 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849676 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llmrp\" (UniqueName: \"kubernetes.io/projected/39fa3c3a-e65b-47e3-85f4-f653277da65c-kube-api-access-llmrp\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.849874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.849874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849732 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.849874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849777 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39fa3c3a-e65b-47e3-85f4-f653277da65c-config-out\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.849874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.849874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849831 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.849874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849881 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39fa3c3a-e65b-47e3-85f4-f653277da65c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-config\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849946 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.849992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850016 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850033 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850055 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850087 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850533 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850133 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850533 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850147 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-web-config\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850533 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850171 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850680 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850651 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850741 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.850741 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850701 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.851255 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850869 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.851255 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.850957 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.851744 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.851503 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.852741 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.852716 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.853666 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.853640 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/39fa3c3a-e65b-47e3-85f4-f653277da65c-config-out\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.853781 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.853668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.854629 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.854585 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-config\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.854722 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.854699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-web-config\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.854815 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.854724 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/39fa3c3a-e65b-47e3-85f4-f653277da65c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.854815 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.854727 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/39fa3c3a-e65b-47e3-85f4-f653277da65c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.855311 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.855281 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.855391 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.855340 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.855486 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.855468 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.855683 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.855668 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.856582 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.856566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.856775 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.856735 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/39fa3c3a-e65b-47e3-85f4-f653277da65c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.859801 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.859743 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llmrp\" (UniqueName: \"kubernetes.io/projected/39fa3c3a-e65b-47e3-85f4-f653277da65c-kube-api-access-llmrp\") pod \"prometheus-k8s-0\" (UID: \"39fa3c3a-e65b-47e3-85f4-f653277da65c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:31.910364 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:31.910336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:24:32.055912 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:32.055816 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 14 18:24:32.058508 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:24:32.058480 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fa3c3a_e65b_47e3_85f4_f653277da65c.slice/crio-0d77c820bae6999bb2d1b4ccfa434e0810b562b0ff66f6ad4cbeb611dbda1507 WatchSource:0}: Error finding container 0d77c820bae6999bb2d1b4ccfa434e0810b562b0ff66f6ad4cbeb611dbda1507: Status 404 returned error can't find the container with id 0d77c820bae6999bb2d1b4ccfa434e0810b562b0ff66f6ad4cbeb611dbda1507 Jul 14 18:24:32.521681 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:32.521652 2580 generic.go:356] "Generic (PLEG): container finished" podID="39fa3c3a-e65b-47e3-85f4-f653277da65c" containerID="bd0660323cee5633f978213b71b28c6e170738a5be47bf72cd6b3db7f3b3cf28" exitCode=0 Jul 14 18:24:32.521837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:32.521720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerDied","Data":"bd0660323cee5633f978213b71b28c6e170738a5be47bf72cd6b3db7f3b3cf28"} Jul 14 18:24:32.521837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:32.521790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"0d77c820bae6999bb2d1b4ccfa434e0810b562b0ff66f6ad4cbeb611dbda1507"} Jul 14 18:24:33.529777 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.529739 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"2bd756098d813f8b8c58c880ef1dfa310cd8cd1c430e92b98f4575aee6bc9d98"} Jul 14 18:24:33.530118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.529851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"df7246ea92f29a7cb5fc69d0e9490560ca20f80bfde70479811cf5fac15c47da"} Jul 14 18:24:33.530118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.529879 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"855bb2fa55829676ae7c11af2db1bd6a2763719c5eb5b9e175b9a41050eb0c59"} Jul 14 18:24:33.530118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.529905 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"49437f448eec6b6c267cb2e7e49757b16a1d935ca6281753783d79d5466daa3e"} Jul 14 18:24:33.530118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.529930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"86a2df8804d8d9bcb69a4cfa6525bf56adb887df9666f7f42b41e2be2656780d"} Jul 14 18:24:33.530118 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.529952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"39fa3c3a-e65b-47e3-85f4-f653277da65c","Type":"ContainerStarted","Data":"d2b9424076366bbb842cd3c534fc17367c0d9f85d9844a94be3caa78cc6da79a"} Jul 14 18:24:33.567468 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:33.567416 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.5492862130000002 podStartE2EDuration="2.56739602s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.109807ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:24:33.565865541 +0000 UTC m=+164.291005601" watchObservedRunningTime="2026-07-14 18:24:33.56739602 +0000 UTC m=+164.292536091" Jul 14 18:24:36.910485 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:24:36.910450 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:25:31.911365 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:25:31.911330 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:25:31.927052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:25:31.927020 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:25:32.751517 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:25:32.751483 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 14 18:26:49.597666 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:26:49.597634 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:26:49.602391 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:26:49.602364 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:26:49.605111 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:26:49.605092 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 18:27:40.663528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.663451 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg"] Jul 14 18:27:40.666623 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.666606 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.758355 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.758320 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 18:27:40.758525 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.758363 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 18:27:40.758525 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.758391 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 18:27:40.758619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.758535 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h5s8h\"" Jul 14 18:27:40.765952 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.765928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b26258e-25bd-4282-a0da-747ebe60e46d-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.766022 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.765958 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b26258e-25bd-4282-a0da-747ebe60e46d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.766022 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.766004 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9pj\" (UniqueName: \"kubernetes.io/projected/7b26258e-25bd-4282-a0da-747ebe60e46d-kube-api-access-4p9pj\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.766141 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.766120 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b26258e-25bd-4282-a0da-747ebe60e46d-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.766200 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.766160 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b26258e-25bd-4282-a0da-747ebe60e46d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.769133 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.769120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 18:27:40.802840 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.802809 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg"] Jul 14 18:27:40.867414 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.867376 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9pj\" (UniqueName: \"kubernetes.io/projected/7b26258e-25bd-4282-a0da-747ebe60e46d-kube-api-access-4p9pj\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.867576 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.867445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b26258e-25bd-4282-a0da-747ebe60e46d-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.867576 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.867465 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b26258e-25bd-4282-a0da-747ebe60e46d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.867711 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.867608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b26258e-25bd-4282-a0da-747ebe60e46d-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.867711 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.867655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b26258e-25bd-4282-a0da-747ebe60e46d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.867980 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.867956 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b26258e-25bd-4282-a0da-747ebe60e46d-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.869893 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.869870 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b26258e-25bd-4282-a0da-747ebe60e46d-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.870010 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.869989 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b26258e-25bd-4282-a0da-747ebe60e46d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.870173 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.870153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b26258e-25bd-4282-a0da-747ebe60e46d-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.885482 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.885459 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9pj\" (UniqueName: \"kubernetes.io/projected/7b26258e-25bd-4282-a0da-747ebe60e46d-kube-api-access-4p9pj\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-2pkvg\" (UID: \"7b26258e-25bd-4282-a0da-747ebe60e46d\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:40.975902 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:40.975836 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:41.195495 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:41.195461 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg"] Jul 14 18:27:41.199626 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:27:41.199601 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b26258e_25bd_4282_a0da_747ebe60e46d.slice/crio-36994d19783c8883f83b1db0a75286b6d19f01de5b0e4c65c44d1ce5f2fcbce1 WatchSource:0}: Error finding container 36994d19783c8883f83b1db0a75286b6d19f01de5b0e4c65c44d1ce5f2fcbce1: Status 404 returned error can't find the container with id 36994d19783c8883f83b1db0a75286b6d19f01de5b0e4c65c44d1ce5f2fcbce1 Jul 14 18:27:41.201300 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:41.201284 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:27:42.172384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:42.172341 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" event={"ID":"7b26258e-25bd-4282-a0da-747ebe60e46d","Type":"ContainerStarted","Data":"36994d19783c8883f83b1db0a75286b6d19f01de5b0e4c65c44d1ce5f2fcbce1"} Jul 14 18:27:45.185654 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:45.185612 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" event={"ID":"7b26258e-25bd-4282-a0da-747ebe60e46d","Type":"ContainerStarted","Data":"29cbfadd8686912febf482d753db2321baaca0f4f2030c2dee9d2e51322d27d8"} Jul 14 18:27:46.190841 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:46.190807 2580 generic.go:356] "Generic (PLEG): container finished" podID="7b26258e-25bd-4282-a0da-747ebe60e46d" containerID="29cbfadd8686912febf482d753db2321baaca0f4f2030c2dee9d2e51322d27d8" exitCode=0 Jul 14 18:27:46.191290 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:46.190867 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" event={"ID":"7b26258e-25bd-4282-a0da-747ebe60e46d","Type":"ContainerDied","Data":"29cbfadd8686912febf482d753db2321baaca0f4f2030c2dee9d2e51322d27d8"} Jul 14 18:27:48.204911 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:48.204870 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" event={"ID":"7b26258e-25bd-4282-a0da-747ebe60e46d","Type":"ContainerStarted","Data":"5bdc2378b19b36fb1e77e566d3e2addf5465611e702160f826e6bd7d3876e4ba"} Jul 14 18:27:48.464382 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:48.464235 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" podStartSLOduration=2.514295653 podStartE2EDuration="8.46421493s" totalImagesPullingTime="4.951520571s" totalInitContainerRuntime="998.398706ms" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:48.463492608 +0000 UTC m=+359.188632685" watchObservedRunningTime="2026-07-14 18:27:48.46421493 +0000 UTC m=+359.189355000" Jul 14 18:27:49.208734 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:49.208705 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:27:59.200421 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.200386 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57"] Jul 14 18:27:59.214686 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.214663 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.222998 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.222970 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 18:27:59.223933 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.223918 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-99wmv\"" Jul 14 18:27:59.224013 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.223935 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 18:27:59.224013 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.224008 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 18:27:59.224123 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.223937 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 18:27:59.224123 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.224044 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:27:59.241304 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.241279 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57"] Jul 14 18:27:59.309561 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.309533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q7lc\" (UniqueName: \"kubernetes.io/projected/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-kube-api-access-9q7lc\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.309690 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.309566 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-manager-config\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.309690 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.309585 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-metrics-cert\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.309813 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.309692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-cert\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.410415 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.410383 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-cert\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.410545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.410426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q7lc\" (UniqueName: \"kubernetes.io/projected/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-kube-api-access-9q7lc\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.410545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.410491 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-manager-config\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.410545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.410524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-metrics-cert\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.411230 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.411207 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-manager-config\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.412999 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.412974 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-metrics-cert\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.413210 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.413192 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-cert\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.420980 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.420961 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q7lc\" (UniqueName: \"kubernetes.io/projected/9d9b4a24-7d39-4ed5-9607-0eeef4a5399c-kube-api-access-9q7lc\") pod \"lws-controller-manager-8c5cc675c-7rv57\" (UID: \"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.523245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.523213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:27:59.667495 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:27:59.667471 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57"] Jul 14 18:27:59.669256 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:27:59.669229 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9b4a24_7d39_4ed5_9607_0eeef4a5399c.slice/crio-34d702c6c3809f1c3341f7fc7832abccd4ec629147909fb88ab417679075c0f6 WatchSource:0}: Error finding container 34d702c6c3809f1c3341f7fc7832abccd4ec629147909fb88ab417679075c0f6: Status 404 returned error can't find the container with id 34d702c6c3809f1c3341f7fc7832abccd4ec629147909fb88ab417679075c0f6 Jul 14 18:28:00.214535 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:00.214506 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-2pkvg" Jul 14 18:28:00.251085 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:00.251048 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" event={"ID":"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c","Type":"ContainerStarted","Data":"34d702c6c3809f1c3341f7fc7832abccd4ec629147909fb88ab417679075c0f6"} Jul 14 18:28:03.263277 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:03.263238 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" event={"ID":"9d9b4a24-7d39-4ed5-9607-0eeef4a5399c","Type":"ContainerStarted","Data":"bc4ae576a3a6cb9616a5d8fdccf744486a41c0086247eb649402f5822b67e940"} Jul 14 18:28:03.263655 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:03.263380 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:28:03.415453 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:03.415407 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" podStartSLOduration=1.872807715 podStartE2EDuration="4.415392423s" totalImagesPullingTime="2.542584708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:03.415203493 +0000 UTC m=+374.140343556" watchObservedRunningTime="2026-07-14 18:28:03.415392423 +0000 UTC m=+374.140532484" Jul 14 18:28:14.268926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:14.268895 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-7rv57" Jul 14 18:28:17.336800 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.336753 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj"] Jul 14 18:28:17.339961 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.339935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343511 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343484 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343611 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343530 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343611 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8klr5\" (UniqueName: \"kubernetes.io/projected/9001966f-6c5f-4cc3-aae6-557d714ca7c2-kube-api-access-8klr5\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343611 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343707 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001966f-6c5f-4cc3-aae6-557d714ca7c2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343739 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343708 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.343800 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.343747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.365259 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.365237 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 14 18:28:17.365371 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.365355 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 14 18:28:17.365444 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.365422 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jgq6l\"" Jul 14 18:28:17.366230 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.366214 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 14 18:28:17.366282 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.366231 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:28:17.366785 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.366772 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:28:17.406450 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.406432 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 14 18:28:17.422913 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.422893 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj"] Jul 14 18:28:17.444948 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.444924 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445029 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.444950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001966f-6c5f-4cc3-aae6-557d714ca7c2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445029 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.444980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445029 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.444995 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445029 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.445012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445216 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.445176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445216 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.445199 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8klr5\" (UniqueName: \"kubernetes.io/projected/9001966f-6c5f-4cc3-aae6-557d714ca7c2-kube-api-access-8klr5\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.445721 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.445676 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.447497 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.447479 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9001966f-6c5f-4cc3-aae6-557d714ca7c2-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.447613 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.447581 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.447657 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.447620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.447908 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.447885 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.503391 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.503357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9001966f-6c5f-4cc3-aae6-557d714ca7c2-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.508496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.508470 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8klr5\" (UniqueName: \"kubernetes.io/projected/9001966f-6c5f-4cc3-aae6-557d714ca7c2-kube-api-access-8klr5\") pod \"istiod-openshift-gateway-94bb8fbfd-qdszj\" (UID: \"9001966f-6c5f-4cc3-aae6-557d714ca7c2\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.649064 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.648984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:17.829270 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:17.829232 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj"] Jul 14 18:28:17.834490 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:28:17.834463 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9001966f_6c5f_4cc3_aae6_557d714ca7c2.slice/crio-42bcab9d857aff02f8abecc7c51c82f97241ab90606d80f723cd5cfd36dbb660 WatchSource:0}: Error finding container 42bcab9d857aff02f8abecc7c51c82f97241ab90606d80f723cd5cfd36dbb660: Status 404 returned error can't find the container with id 42bcab9d857aff02f8abecc7c51c82f97241ab90606d80f723cd5cfd36dbb660 Jul 14 18:28:18.317108 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:18.317072 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" event={"ID":"9001966f-6c5f-4cc3-aae6-557d714ca7c2","Type":"ContainerStarted","Data":"42bcab9d857aff02f8abecc7c51c82f97241ab90606d80f723cd5cfd36dbb660"} Jul 14 18:28:21.553503 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:21.553462 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:28:21.553735 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:21.553545 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:28:22.335369 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:22.335328 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" event={"ID":"9001966f-6c5f-4cc3-aae6-557d714ca7c2","Type":"ContainerStarted","Data":"8e33ff0c195c8aecd6ad02e6d14a7ceeed6554b5ecc2a226deaab0f55c0bda25"} Jul 14 18:28:22.335602 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:22.335579 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:28:22.337339 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:22.337310 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-qdszj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:28:22.337445 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:22.337361 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" podUID="9001966f-6c5f-4cc3-aae6-557d714ca7c2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:28:22.442572 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:22.442526 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" podStartSLOduration=1.725658366 podStartE2EDuration="5.442512262s" totalImagesPullingTime="3.716853896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:22.442054468 +0000 UTC m=+393.167194530" watchObservedRunningTime="2026-07-14 18:28:22.442512262 +0000 UTC m=+393.167652322" Jul 14 18:28:23.339906 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:28:23.339866 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-qdszj" Jul 14 18:29:50.456775 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.456726 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4"] Jul 14 18:29:50.459825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.459809 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.464182 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.464162 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:29:50.464889 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.464866 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:29:50.468637 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.468612 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r57vd\"" Jul 14 18:29:50.490533 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.490504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4"] Jul 14 18:29:50.626286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.626246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q48t\" (UniqueName: \"kubernetes.io/projected/bcc6c621-7b50-41d0-95d0-da334311ae85-kube-api-access-5q48t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.626286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.626293 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcc6c621-7b50-41d0-95d0-da334311ae85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.726668 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.726573 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcc6c621-7b50-41d0-95d0-da334311ae85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.726668 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.726648 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q48t\" (UniqueName: \"kubernetes.io/projected/bcc6c621-7b50-41d0-95d0-da334311ae85-kube-api-access-5q48t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.726991 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.726965 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcc6c621-7b50-41d0-95d0-da334311ae85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.739040 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.739018 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q48t\" (UniqueName: \"kubernetes.io/projected/bcc6c621-7b50-41d0-95d0-da334311ae85-kube-api-access-5q48t\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.769837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.769802 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:50.936270 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:50.936238 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4"] Jul 14 18:29:50.940571 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:29:50.940540 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc6c621_7b50_41d0_95d0_da334311ae85.slice/crio-a935aa6a8ff8506b9acb0d879f1aeff9127a641fd45f2e87b3b89e4e7a478b23 WatchSource:0}: Error finding container a935aa6a8ff8506b9acb0d879f1aeff9127a641fd45f2e87b3b89e4e7a478b23: Status 404 returned error can't find the container with id a935aa6a8ff8506b9acb0d879f1aeff9127a641fd45f2e87b3b89e4e7a478b23 Jul 14 18:29:51.651221 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:51.651185 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" event={"ID":"bcc6c621-7b50-41d0-95d0-da334311ae85","Type":"ContainerStarted","Data":"a935aa6a8ff8506b9acb0d879f1aeff9127a641fd45f2e87b3b89e4e7a478b23"} Jul 14 18:29:59.689803 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:59.689701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" event={"ID":"bcc6c621-7b50-41d0-95d0-da334311ae85","Type":"ContainerStarted","Data":"99ea0eb2e5ca998c3cdf0628414f569cfb7f4c4704beb5c01c8566bdae12aa36"} Jul 14 18:29:59.690262 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:59.689810 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:29:59.747214 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:29:59.747167 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" podStartSLOduration=1.262668282 podStartE2EDuration="9.747152605s" totalImagesPullingTime="8.484484323s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:29:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:29:59.745316494 +0000 UTC m=+490.470456555" watchObservedRunningTime="2026-07-14 18:29:59.747152605 +0000 UTC m=+490.472292666" Jul 14 18:30:10.695979 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:10.695785 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:12.562700 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.562668 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4"] Jul 14 18:30:12.563079 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.562884 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" containerName="manager" containerID="cri-o://99ea0eb2e5ca998c3cdf0628414f569cfb7f4c4704beb5c01c8566bdae12aa36" gracePeriod=2 Jul 14 18:30:12.577397 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.577368 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4"] Jul 14 18:30:12.578290 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:30:12.578256 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"bcc6c621-7b50-41d0-95d0-da334311ae85\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\": pods \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:12.598208 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.598178 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56"] Jul 14 18:30:12.598663 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.598643 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" containerName="manager" Jul 14 18:30:12.598731 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.598668 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" containerName="manager" Jul 14 18:30:12.598809 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.598779 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" containerName="manager" Jul 14 18:30:12.601747 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.601725 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.604006 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:30:12.603980 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" is forbidden: User \"system:node:ip-10-0-131-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-199.ec2.internal' and this object" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:12.627263 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.627235 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56"] Jul 14 18:30:12.693079 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.693051 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffd7570-b00c-4e63-83d6-12e963ec3727-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ffx56\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.693230 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.693089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/5ffd7570-b00c-4e63-83d6-12e963ec3727-kube-api-access-fw87b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ffx56\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.743837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.743799 2580 generic.go:356] "Generic (PLEG): container finished" podID="bcc6c621-7b50-41d0-95d0-da334311ae85" containerID="99ea0eb2e5ca998c3cdf0628414f569cfb7f4c4704beb5c01c8566bdae12aa36" exitCode=0 Jul 14 18:30:12.794421 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.794398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffd7570-b00c-4e63-83d6-12e963ec3727-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ffx56\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.794537 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.794462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/5ffd7570-b00c-4e63-83d6-12e963ec3727-kube-api-access-fw87b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ffx56\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.794840 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.794822 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffd7570-b00c-4e63-83d6-12e963ec3727-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ffx56\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.801088 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.801070 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:12.809482 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:30:12.809452 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" is forbidden: User \"system:node:ip-10-0-131-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-199.ec2.internal' and this object" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:12.811726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.811712 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/5ffd7570-b00c-4e63-83d6-12e963ec3727-kube-api-access-fw87b\") pod \"kuadrant-operator-controller-manager-8684c9ff49-ffx56\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.894985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.894929 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcc6c621-7b50-41d0-95d0-da334311ae85-extensions-socket-volume\") pod \"bcc6c621-7b50-41d0-95d0-da334311ae85\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " Jul 14 18:30:12.894985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.894981 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5q48t\" (UniqueName: \"kubernetes.io/projected/bcc6c621-7b50-41d0-95d0-da334311ae85-kube-api-access-5q48t\") pod \"bcc6c621-7b50-41d0-95d0-da334311ae85\" (UID: \"bcc6c621-7b50-41d0-95d0-da334311ae85\") " Jul 14 18:30:12.895368 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.895347 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcc6c621-7b50-41d0-95d0-da334311ae85-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "bcc6c621-7b50-41d0-95d0-da334311ae85" (UID: "bcc6c621-7b50-41d0-95d0-da334311ae85"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:30:12.897084 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.897064 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc6c621-7b50-41d0-95d0-da334311ae85-kube-api-access-5q48t" (OuterVolumeSpecName: "kube-api-access-5q48t") pod "bcc6c621-7b50-41d0-95d0-da334311ae85" (UID: "bcc6c621-7b50-41d0-95d0-da334311ae85"). InnerVolumeSpecName "kube-api-access-5q48t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:30:12.941398 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.941371 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:12.996159 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.996126 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bcc6c621-7b50-41d0-95d0-da334311ae85-extensions-socket-volume\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:30:12.996293 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:12.996166 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q48t\" (UniqueName: \"kubernetes.io/projected/bcc6c621-7b50-41d0-95d0-da334311ae85-kube-api-access-5q48t\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:30:13.095706 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.095672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56"] Jul 14 18:30:13.100333 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:30:13.100311 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ffd7570_b00c_4e63_83d6_12e963ec3727.slice/crio-94e7bcf3b57539261f48924b16e6f6a28fcb4deede03f300bb1e2ee8707fba37 WatchSource:0}: Error finding container 94e7bcf3b57539261f48924b16e6f6a28fcb4deede03f300bb1e2ee8707fba37: Status 404 returned error can't find the container with id 94e7bcf3b57539261f48924b16e6f6a28fcb4deede03f300bb1e2ee8707fba37 Jul 14 18:30:13.749157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.749127 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:13.749157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.749143 2580 scope.go:117] "RemoveContainer" containerID="99ea0eb2e5ca998c3cdf0628414f569cfb7f4c4704beb5c01c8566bdae12aa36" Jul 14 18:30:13.750917 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.750892 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" event={"ID":"5ffd7570-b00c-4e63-83d6-12e963ec3727","Type":"ContainerStarted","Data":"c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370"} Jul 14 18:30:13.751023 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.750938 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" event={"ID":"5ffd7570-b00c-4e63-83d6-12e963ec3727","Type":"ContainerStarted","Data":"94e7bcf3b57539261f48924b16e6f6a28fcb4deede03f300bb1e2ee8707fba37"} Jul 14 18:30:13.751023 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.750988 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:13.752703 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:30:13.752678 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" is forbidden: User \"system:node:ip-10-0-131-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-199.ec2.internal' and this object" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:13.773462 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:30:13.773433 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-qbrf4\" is forbidden: User \"system:node:ip-10-0-131-199.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-199.ec2.internal' and this object" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qbrf4" Jul 14 18:30:13.790308 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.790280 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcc6c621-7b50-41d0-95d0-da334311ae85" path="/var/lib/kubelet/pods/bcc6c621-7b50-41d0-95d0-da334311ae85/volumes" Jul 14 18:30:13.835588 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:13.835539 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" podStartSLOduration=1.835525361 podStartE2EDuration="1.835525361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:13.834223042 +0000 UTC m=+504.559363103" watchObservedRunningTime="2026-07-14 18:30:13.835525361 +0000 UTC m=+504.560665422" Jul 14 18:30:24.758537 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:24.758508 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:28.941266 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:28.941226 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56"] Jul 14 18:30:28.941717 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:28.941452 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" podUID="5ffd7570-b00c-4e63-83d6-12e963ec3727" containerName="manager" containerID="cri-o://c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370" gracePeriod=10 Jul 14 18:30:29.178798 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.178773 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:29.187310 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.187288 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n"] Jul 14 18:30:29.187561 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.187547 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ffd7570-b00c-4e63-83d6-12e963ec3727" containerName="manager" Jul 14 18:30:29.187611 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.187563 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ffd7570-b00c-4e63-83d6-12e963ec3727" containerName="manager" Jul 14 18:30:29.187653 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.187612 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ffd7570-b00c-4e63-83d6-12e963ec3727" containerName="manager" Jul 14 18:30:29.189963 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.189948 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.207671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.207649 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n"] Jul 14 18:30:29.228160 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.228138 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffd7570-b00c-4e63-83d6-12e963ec3727-extensions-socket-volume\") pod \"5ffd7570-b00c-4e63-83d6-12e963ec3727\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " Jul 14 18:30:29.228267 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.228182 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/5ffd7570-b00c-4e63-83d6-12e963ec3727-kube-api-access-fw87b\") pod \"5ffd7570-b00c-4e63-83d6-12e963ec3727\" (UID: \"5ffd7570-b00c-4e63-83d6-12e963ec3727\") " Jul 14 18:30:29.228514 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.228494 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ffd7570-b00c-4e63-83d6-12e963ec3727-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5ffd7570-b00c-4e63-83d6-12e963ec3727" (UID: "5ffd7570-b00c-4e63-83d6-12e963ec3727"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:30:29.230368 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.230345 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ffd7570-b00c-4e63-83d6-12e963ec3727-kube-api-access-fw87b" (OuterVolumeSpecName: "kube-api-access-fw87b") pod "5ffd7570-b00c-4e63-83d6-12e963ec3727" (UID: "5ffd7570-b00c-4e63-83d6-12e963ec3727"). InnerVolumeSpecName "kube-api-access-fw87b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:30:29.329517 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.329485 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wglb8\" (UniqueName: \"kubernetes.io/projected/39f21b31-eafd-4e93-a058-b62b038bd82a-kube-api-access-wglb8\") pod \"kuadrant-operator-controller-manager-86878c769f-l4b5n\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.329659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.329532 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f21b31-eafd-4e93-a058-b62b038bd82a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l4b5n\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.329659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.329616 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ffd7570-b00c-4e63-83d6-12e963ec3727-extensions-socket-volume\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:30:29.329659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.329635 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw87b\" (UniqueName: \"kubernetes.io/projected/5ffd7570-b00c-4e63-83d6-12e963ec3727-kube-api-access-fw87b\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:30:29.430936 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.430904 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglb8\" (UniqueName: \"kubernetes.io/projected/39f21b31-eafd-4e93-a058-b62b038bd82a-kube-api-access-wglb8\") pod \"kuadrant-operator-controller-manager-86878c769f-l4b5n\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.431078 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.430955 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f21b31-eafd-4e93-a058-b62b038bd82a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l4b5n\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.431245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.431229 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f21b31-eafd-4e93-a058-b62b038bd82a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-l4b5n\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.446465 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.446402 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglb8\" (UniqueName: \"kubernetes.io/projected/39f21b31-eafd-4e93-a058-b62b038bd82a-kube-api-access-wglb8\") pod \"kuadrant-operator-controller-manager-86878c769f-l4b5n\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.498909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.498883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.628701 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.628668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n"] Jul 14 18:30:29.632852 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:30:29.632819 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f21b31_eafd_4e93_a058_b62b038bd82a.slice/crio-5169c99c31b00cfb9c7ad4b11ebb6849c8d46360eb89dfec17b1a5d6b9421311 WatchSource:0}: Error finding container 5169c99c31b00cfb9c7ad4b11ebb6849c8d46360eb89dfec17b1a5d6b9421311: Status 404 returned error can't find the container with id 5169c99c31b00cfb9c7ad4b11ebb6849c8d46360eb89dfec17b1a5d6b9421311 Jul 14 18:30:29.818256 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.818226 2580 generic.go:356] "Generic (PLEG): container finished" podID="5ffd7570-b00c-4e63-83d6-12e963ec3727" containerID="c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370" exitCode=0 Jul 14 18:30:29.818400 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.818300 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" Jul 14 18:30:29.818400 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.818316 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" event={"ID":"5ffd7570-b00c-4e63-83d6-12e963ec3727","Type":"ContainerDied","Data":"c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370"} Jul 14 18:30:29.818400 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.818369 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56" event={"ID":"5ffd7570-b00c-4e63-83d6-12e963ec3727","Type":"ContainerDied","Data":"94e7bcf3b57539261f48924b16e6f6a28fcb4deede03f300bb1e2ee8707fba37"} Jul 14 18:30:29.818400 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.818396 2580 scope.go:117] "RemoveContainer" containerID="c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370" Jul 14 18:30:29.820448 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.820422 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" event={"ID":"39f21b31-eafd-4e93-a058-b62b038bd82a","Type":"ContainerStarted","Data":"2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e"} Jul 14 18:30:29.820557 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.820473 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" event={"ID":"39f21b31-eafd-4e93-a058-b62b038bd82a","Type":"ContainerStarted","Data":"5169c99c31b00cfb9c7ad4b11ebb6849c8d46360eb89dfec17b1a5d6b9421311"} Jul 14 18:30:29.820621 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.820600 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:30:29.828281 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.828269 2580 scope.go:117] "RemoveContainer" containerID="c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370" Jul 14 18:30:29.828558 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:30:29.828542 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370\": container with ID starting with c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370 not found: ID does not exist" containerID="c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370" Jul 14 18:30:29.828617 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.828566 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370"} err="failed to get container status \"c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370\": rpc error: code = NotFound desc = could not find container \"c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370\": container with ID starting with c124c25d35de69ef6154a5d0bd16198283af8ab05883926732ec5cc40876b370 not found: ID does not exist" Jul 14 18:30:29.843943 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.843919 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56"] Jul 14 18:30:29.845960 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.845938 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-ffx56"] Jul 14 18:30:29.883989 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:29.883951 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" podStartSLOduration=0.883937404 podStartE2EDuration="883.937404ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:29.882297472 +0000 UTC m=+520.607437532" watchObservedRunningTime="2026-07-14 18:30:29.883937404 +0000 UTC m=+520.609077466" Jul 14 18:30:31.783016 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:31.782983 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ffd7570-b00c-4e63-83d6-12e963ec3727" path="/var/lib/kubelet/pods/5ffd7570-b00c-4e63-83d6-12e963ec3727/volumes" Jul 14 18:30:40.827230 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:30:40.827151 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:31:42.142021 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.141985 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-846475b4dc-z8ktw"] Jul 14 18:31:42.145267 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.145251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.155775 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.155739 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ws7xn\"" Jul 14 18:31:42.156978 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.156961 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 14 18:31:42.164332 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.164317 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 14 18:31:42.187996 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.187975 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-846475b4dc-z8ktw"] Jul 14 18:31:42.192239 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.192220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ddw6\" (UniqueName: \"kubernetes.io/projected/da6ad72e-02c8-4fa1-beda-d207f938a7bc-kube-api-access-9ddw6\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.192339 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.192260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.292657 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.292622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ddw6\" (UniqueName: \"kubernetes.io/projected/da6ad72e-02c8-4fa1-beda-d207f938a7bc-kube-api-access-9ddw6\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.292877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.292675 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.292877 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:31:42.292792 2580 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 14 18:31:42.292877 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:31:42.292843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert podName:da6ad72e-02c8-4fa1-beda-d207f938a7bc nodeName:}" failed. No retries permitted until 2026-07-14 18:31:42.79282827 +0000 UTC m=+593.517968311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert") pod "maas-controller-846475b4dc-z8ktw" (UID: "da6ad72e-02c8-4fa1-beda-d207f938a7bc") : secret "maas-controller-webhook-cert" not found Jul 14 18:31:42.308652 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.308629 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ddw6\" (UniqueName: \"kubernetes.io/projected/da6ad72e-02c8-4fa1-beda-d207f938a7bc-kube-api-access-9ddw6\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.796877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.796846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:42.799101 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:42.799075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert\") pod \"maas-controller-846475b4dc-z8ktw\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:43.055406 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:43.055332 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:43.196245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:43.196215 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-846475b4dc-z8ktw"] Jul 14 18:31:43.200401 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:31:43.200375 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ad72e_02c8_4fa1_beda_d207f938a7bc.slice/crio-ae239a494531cf90690903e7e69265697090eee2c6ade77de5d077f9ba616a2c WatchSource:0}: Error finding container ae239a494531cf90690903e7e69265697090eee2c6ade77de5d077f9ba616a2c: Status 404 returned error can't find the container with id ae239a494531cf90690903e7e69265697090eee2c6ade77de5d077f9ba616a2c Jul 14 18:31:44.108336 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:44.108281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-846475b4dc-z8ktw" event={"ID":"da6ad72e-02c8-4fa1-beda-d207f938a7bc","Type":"ContainerStarted","Data":"ae239a494531cf90690903e7e69265697090eee2c6ade77de5d077f9ba616a2c"} Jul 14 18:31:46.118651 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:46.118617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-846475b4dc-z8ktw" event={"ID":"da6ad72e-02c8-4fa1-beda-d207f938a7bc","Type":"ContainerStarted","Data":"a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9"} Jul 14 18:31:46.119083 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:46.118749 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:31:46.215744 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:46.215694 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-846475b4dc-z8ktw" podStartSLOduration=1.746469107 podStartE2EDuration="4.215676715s" totalImagesPullingTime="2.469207608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:46.21308237 +0000 UTC m=+596.938222431" watchObservedRunningTime="2026-07-14 18:31:46.215676715 +0000 UTC m=+596.940816777" Jul 14 18:31:49.621887 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:49.621860 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:31:49.627269 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:49.627248 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:31:57.124549 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:31:57.124519 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:32:19.002824 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.002778 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk"] Jul 14 18:32:19.006114 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.006097 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.010167 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.010145 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 14 18:32:19.010880 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.010863 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-x52zq\"" Jul 14 18:32:19.018683 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.018658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk"] Jul 14 18:32:19.086901 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.086867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b7bc-64bc-4e4c-8d11-07e6f3d310d5-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5kgtk\" (UID: \"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.086901 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.086907 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtgwq\" (UniqueName: \"kubernetes.io/projected/5844b7bc-64bc-4e4c-8d11-07e6f3d310d5-kube-api-access-wtgwq\") pod \"payload-pre-processing-74584bb9f8-5kgtk\" (UID: \"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.187610 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.187577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b7bc-64bc-4e4c-8d11-07e6f3d310d5-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5kgtk\" (UID: \"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.187610 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.187615 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtgwq\" (UniqueName: \"kubernetes.io/projected/5844b7bc-64bc-4e4c-8d11-07e6f3d310d5-kube-api-access-wtgwq\") pod \"payload-pre-processing-74584bb9f8-5kgtk\" (UID: \"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.188277 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.188256 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5844b7bc-64bc-4e4c-8d11-07e6f3d310d5-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-5kgtk\" (UID: \"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.203608 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.203579 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtgwq\" (UniqueName: \"kubernetes.io/projected/5844b7bc-64bc-4e4c-8d11-07e6f3d310d5-kube-api-access-wtgwq\") pod \"payload-pre-processing-74584bb9f8-5kgtk\" (UID: \"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.316428 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.316398 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:19.454104 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:19.454075 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk"] Jul 14 18:32:19.456505 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:32:19.456478 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5844b7bc_64bc_4e4c_8d11_07e6f3d310d5.slice/crio-935fc0184d9c803b42d6ac73aa10c9d9939b20e8105ed9e032caec8fcd2cb07b WatchSource:0}: Error finding container 935fc0184d9c803b42d6ac73aa10c9d9939b20e8105ed9e032caec8fcd2cb07b: Status 404 returned error can't find the container with id 935fc0184d9c803b42d6ac73aa10c9d9939b20e8105ed9e032caec8fcd2cb07b Jul 14 18:32:20.250629 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:20.250596 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" event={"ID":"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5","Type":"ContainerStarted","Data":"935fc0184d9c803b42d6ac73aa10c9d9939b20e8105ed9e032caec8fcd2cb07b"} Jul 14 18:32:22.259491 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:22.259457 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" event={"ID":"5844b7bc-64bc-4e4c-8d11-07e6f3d310d5","Type":"ContainerStarted","Data":"b0e9ba289c6412a42a35731f6cab79206219770fe3f6275f091b11d4445f02ae"} Jul 14 18:32:22.259899 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:22.259577 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:33.266432 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:33.266395 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" Jul 14 18:32:33.290257 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:33.290198 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-5kgtk" podStartSLOduration=13.461266003 podStartE2EDuration="15.290176965s" totalImagesPullingTime="1.828910962s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:32:22.278485673 +0000 UTC m=+633.003625734" watchObservedRunningTime="2026-07-14 18:32:33.290176965 +0000 UTC m=+644.015317027" Jul 14 18:32:45.481975 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.481945 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg"] Jul 14 18:32:45.485173 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.485153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.487878 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.487858 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 18:32:45.488868 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.488850 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 18:32:45.488975 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.488862 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 14 18:32:45.488975 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.488911 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bqqxv\"" Jul 14 18:32:45.504638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.504602 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg"] Jul 14 18:32:45.593545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.593511 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.593726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.593551 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.593726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.593601 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clfnt\" (UniqueName: \"kubernetes.io/projected/30ea3f47-8521-47bd-95f9-30811d34b616-kube-api-access-clfnt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.593726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.593633 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.593726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.593692 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30ea3f47-8521-47bd-95f9-30811d34b616-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.593726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.593714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.694863 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.694827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.694870 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.694919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clfnt\" (UniqueName: \"kubernetes.io/projected/30ea3f47-8521-47bd-95f9-30811d34b616-kube-api-access-clfnt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.694950 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.694990 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30ea3f47-8521-47bd-95f9-30811d34b616-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695232 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.695116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695340 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.695320 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695429 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.695410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.695477 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.695455 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.697245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.697225 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30ea3f47-8521-47bd-95f9-30811d34b616-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.697498 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.697480 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30ea3f47-8521-47bd-95f9-30811d34b616-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.704420 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.704394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clfnt\" (UniqueName: \"kubernetes.io/projected/30ea3f47-8521-47bd-95f9-30811d34b616-kube-api-access-clfnt\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg\" (UID: \"30ea3f47-8521-47bd-95f9-30811d34b616\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.795178 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.795148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:45.943140 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.943114 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg"] Jul 14 18:32:45.945100 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:32:45.945078 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ea3f47_8521_47bd_95f9_30811d34b616.slice/crio-a3549432779b9a8e59f9c0265107628c75c3bc98eb84a2297d8c41f3601c0153 WatchSource:0}: Error finding container a3549432779b9a8e59f9c0265107628c75c3bc98eb84a2297d8c41f3601c0153: Status 404 returned error can't find the container with id a3549432779b9a8e59f9c0265107628c75c3bc98eb84a2297d8c41f3601c0153 Jul 14 18:32:45.946634 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:45.946613 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:32:46.364538 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:46.364491 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" event={"ID":"30ea3f47-8521-47bd-95f9-30811d34b616","Type":"ContainerStarted","Data":"a3549432779b9a8e59f9c0265107628c75c3bc98eb84a2297d8c41f3601c0153"} Jul 14 18:32:47.370969 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:47.370929 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" event={"ID":"30ea3f47-8521-47bd-95f9-30811d34b616","Type":"ContainerStarted","Data":"de94b8e80c9cb3d5d8ab5a2f8a60caa4f014927ade7cbca366ec3b00896cdd5c"} Jul 14 18:32:47.395973 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:47.395924 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" podStartSLOduration=1.177114014 podStartE2EDuration="2.395908715s" totalImagesPullingTime="1.218794701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:32:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:32:47.394827964 +0000 UTC m=+658.119968026" watchObservedRunningTime="2026-07-14 18:32:47.395908715 +0000 UTC m=+658.121048776" Jul 14 18:32:55.795405 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:55.795367 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:55.795821 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:55.795660 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:55.807866 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:55.807844 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:32:56.419825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:32:56.419796 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg" Jul 14 18:33:01.691285 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.691249 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7"] Jul 14 18:33:01.694745 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.694723 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.709954 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.709931 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 14 18:33:01.718027 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.718000 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7"] Jul 14 18:33:01.834696 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.834667 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.834884 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.834707 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.834884 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.834729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08dd9b45-aed8-4cb1-be81-696a353071a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.834884 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.834831 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mk5\" (UniqueName: \"kubernetes.io/projected/08dd9b45-aed8-4cb1-be81-696a353071a9-kube-api-access-m4mk5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.834994 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.834894 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.834994 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.834946 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935484 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mk5\" (UniqueName: \"kubernetes.io/projected/08dd9b45-aed8-4cb1-be81-696a353071a9-kube-api-access-m4mk5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935484 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935695 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935695 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935557 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935695 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935578 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935695 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08dd9b45-aed8-4cb1-be81-696a353071a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.935940 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935920 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.936002 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.935982 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.936078 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.936049 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.937848 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.937829 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08dd9b45-aed8-4cb1-be81-696a353071a9-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.938122 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.938103 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08dd9b45-aed8-4cb1-be81-696a353071a9-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:01.955941 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:01.955874 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mk5\" (UniqueName: \"kubernetes.io/projected/08dd9b45-aed8-4cb1-be81-696a353071a9-kube-api-access-m4mk5\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7\" (UID: \"08dd9b45-aed8-4cb1-be81-696a353071a9\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:02.004821 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:02.004792 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:02.163625 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:02.163584 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7"] Jul 14 18:33:02.165714 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:33:02.165686 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dd9b45_aed8_4cb1_be81_696a353071a9.slice/crio-f403ddbbca600367c978607ea86129a22d9f6d4ed421ecf6b73578623b7831d5 WatchSource:0}: Error finding container f403ddbbca600367c978607ea86129a22d9f6d4ed421ecf6b73578623b7831d5: Status 404 returned error can't find the container with id f403ddbbca600367c978607ea86129a22d9f6d4ed421ecf6b73578623b7831d5 Jul 14 18:33:02.441090 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:02.441054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" event={"ID":"08dd9b45-aed8-4cb1-be81-696a353071a9","Type":"ContainerStarted","Data":"0b09a04244a5563594dbd9932229a7832503d2d1efce6f70011cddd6f3af0c26"} Jul 14 18:33:02.441229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:02.441112 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" event={"ID":"08dd9b45-aed8-4cb1-be81-696a353071a9","Type":"ContainerStarted","Data":"f403ddbbca600367c978607ea86129a22d9f6d4ed421ecf6b73578623b7831d5"} Jul 14 18:33:12.006016 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:12.005984 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:12.006449 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:12.006042 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:12.018352 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:12.018330 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:33:12.051271 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:12.051225 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" podStartSLOduration=10.872110208 podStartE2EDuration="11.05120964s" totalImagesPullingTime="179.099432ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:33:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:33:02.473097878 +0000 UTC m=+673.198237939" watchObservedRunningTime="2026-07-14 18:33:12.05120964 +0000 UTC m=+682.776349700" Jul 14 18:33:12.492655 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:33:12.492630 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7" Jul 14 18:35:44.681539 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:44.681507 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-846475b4dc-z8ktw"] Jul 14 18:35:44.682113 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:44.681794 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-846475b4dc-z8ktw" podUID="da6ad72e-02c8-4fa1-beda-d207f938a7bc" containerName="manager" containerID="cri-o://a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9" gracePeriod=30 Jul 14 18:35:44.921091 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:44.921067 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:35:45.072670 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.072639 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ddw6\" (UniqueName: \"kubernetes.io/projected/da6ad72e-02c8-4fa1-beda-d207f938a7bc-kube-api-access-9ddw6\") pod \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " Jul 14 18:35:45.072823 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.072715 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert\") pod \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\" (UID: \"da6ad72e-02c8-4fa1-beda-d207f938a7bc\") " Jul 14 18:35:45.074915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.074884 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da6ad72e-02c8-4fa1-beda-d207f938a7bc-kube-api-access-9ddw6" (OuterVolumeSpecName: "kube-api-access-9ddw6") pod "da6ad72e-02c8-4fa1-beda-d207f938a7bc" (UID: "da6ad72e-02c8-4fa1-beda-d207f938a7bc"). InnerVolumeSpecName "kube-api-access-9ddw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:35:45.074915 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.074909 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "da6ad72e-02c8-4fa1-beda-d207f938a7bc" (UID: "da6ad72e-02c8-4fa1-beda-d207f938a7bc"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:35:45.104084 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.104052 2580 generic.go:356] "Generic (PLEG): container finished" podID="da6ad72e-02c8-4fa1-beda-d207f938a7bc" containerID="a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9" exitCode=0 Jul 14 18:35:45.104195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.104102 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-846475b4dc-z8ktw" Jul 14 18:35:45.104195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.104117 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-846475b4dc-z8ktw" event={"ID":"da6ad72e-02c8-4fa1-beda-d207f938a7bc","Type":"ContainerDied","Data":"a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9"} Jul 14 18:35:45.104195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.104158 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-846475b4dc-z8ktw" event={"ID":"da6ad72e-02c8-4fa1-beda-d207f938a7bc","Type":"ContainerDied","Data":"ae239a494531cf90690903e7e69265697090eee2c6ade77de5d077f9ba616a2c"} Jul 14 18:35:45.104195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.104178 2580 scope.go:117] "RemoveContainer" containerID="a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9" Jul 14 18:35:45.112724 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.112706 2580 scope.go:117] "RemoveContainer" containerID="a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9" Jul 14 18:35:45.112994 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:35:45.112976 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9\": container with ID starting with a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9 not found: ID does not exist" containerID="a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9" Jul 14 18:35:45.113051 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.113002 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9"} err="failed to get container status \"a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9\": rpc error: code = NotFound desc = could not find container \"a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9\": container with ID starting with a5c3ae3bb3860b5ab9191fefe1527ad5f12c989f4e9ead65f0b25baf90f781d9 not found: ID does not exist" Jul 14 18:35:45.125052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.125028 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-846475b4dc-z8ktw"] Jul 14 18:35:45.129820 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.129795 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-846475b4dc-z8ktw"] Jul 14 18:35:45.173805 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.173755 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/da6ad72e-02c8-4fa1-beda-d207f938a7bc-webhook-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.173805 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.173803 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ddw6\" (UniqueName: \"kubernetes.io/projected/da6ad72e-02c8-4fa1-beda-d207f938a7bc-kube-api-access-9ddw6\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:35:45.783089 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:35:45.783052 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6ad72e-02c8-4fa1-beda-d207f938a7bc" path="/var/lib/kubelet/pods/da6ad72e-02c8-4fa1-beda-d207f938a7bc/volumes" Jul 14 18:36:49.647643 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:36:49.647559 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:36:49.650347 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:36:49.650324 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:40:44.012810 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:44.012742 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n"] Jul 14 18:40:44.015646 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:44.013099 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" podUID="39f21b31-eafd-4e93-a058-b62b038bd82a" containerName="manager" containerID="cri-o://2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e" gracePeriod=10 Jul 14 18:40:46.767732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:46.767702 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:40:46.913320 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:46.913240 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f21b31-eafd-4e93-a058-b62b038bd82a-extensions-socket-volume\") pod \"39f21b31-eafd-4e93-a058-b62b038bd82a\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " Jul 14 18:40:46.913464 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:46.913352 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wglb8\" (UniqueName: \"kubernetes.io/projected/39f21b31-eafd-4e93-a058-b62b038bd82a-kube-api-access-wglb8\") pod \"39f21b31-eafd-4e93-a058-b62b038bd82a\" (UID: \"39f21b31-eafd-4e93-a058-b62b038bd82a\") " Jul 14 18:40:46.913694 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:46.913661 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39f21b31-eafd-4e93-a058-b62b038bd82a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "39f21b31-eafd-4e93-a058-b62b038bd82a" (UID: "39f21b31-eafd-4e93-a058-b62b038bd82a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:40:46.915492 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:46.915468 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f21b31-eafd-4e93-a058-b62b038bd82a-kube-api-access-wglb8" (OuterVolumeSpecName: "kube-api-access-wglb8") pod "39f21b31-eafd-4e93-a058-b62b038bd82a" (UID: "39f21b31-eafd-4e93-a058-b62b038bd82a"). InnerVolumeSpecName "kube-api-access-wglb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:40:47.014626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.014590 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wglb8\" (UniqueName: \"kubernetes.io/projected/39f21b31-eafd-4e93-a058-b62b038bd82a-kube-api-access-wglb8\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:40:47.014626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.014620 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39f21b31-eafd-4e93-a058-b62b038bd82a-extensions-socket-volume\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:40:47.339248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.339208 2580 generic.go:356] "Generic (PLEG): container finished" podID="39f21b31-eafd-4e93-a058-b62b038bd82a" containerID="2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e" exitCode=0 Jul 14 18:40:47.339248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.339246 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" Jul 14 18:40:47.339463 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.339257 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" event={"ID":"39f21b31-eafd-4e93-a058-b62b038bd82a","Type":"ContainerDied","Data":"2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e"} Jul 14 18:40:47.339463 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.339313 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n" event={"ID":"39f21b31-eafd-4e93-a058-b62b038bd82a","Type":"ContainerDied","Data":"5169c99c31b00cfb9c7ad4b11ebb6849c8d46360eb89dfec17b1a5d6b9421311"} Jul 14 18:40:47.339463 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.339345 2580 scope.go:117] "RemoveContainer" containerID="2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e" Jul 14 18:40:47.350728 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.350702 2580 scope.go:117] "RemoveContainer" containerID="2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e" Jul 14 18:40:47.351271 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:40:47.351237 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e\": container with ID starting with 2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e not found: ID does not exist" containerID="2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e" Jul 14 18:40:47.351364 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.351269 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e"} err="failed to get container status \"2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e\": rpc error: code = NotFound desc = could not find container \"2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e\": container with ID starting with 2a1b4358c317be1add25868ec7d8d6a53a484ebed86965edd1e14303e0d2069e not found: ID does not exist" Jul 14 18:40:47.362723 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.362696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n"] Jul 14 18:40:47.366058 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.366035 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-l4b5n"] Jul 14 18:40:47.782510 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:40:47.782483 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39f21b31-eafd-4e93-a058-b62b038bd82a" path="/var/lib/kubelet/pods/39f21b31-eafd-4e93-a058-b62b038bd82a/volumes" Jul 14 18:41:45.289693 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.289646 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl"] Jul 14 18:41:45.290251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.289968 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39f21b31-eafd-4e93-a058-b62b038bd82a" containerName="manager" Jul 14 18:41:45.290251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.289979 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="39f21b31-eafd-4e93-a058-b62b038bd82a" containerName="manager" Jul 14 18:41:45.290251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.289991 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ad72e-02c8-4fa1-beda-d207f938a7bc" containerName="manager" Jul 14 18:41:45.290251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.289998 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ad72e-02c8-4fa1-beda-d207f938a7bc" containerName="manager" Jul 14 18:41:45.290251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.290045 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ad72e-02c8-4fa1-beda-d207f938a7bc" containerName="manager" Jul 14 18:41:45.290251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.290052 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="39f21b31-eafd-4e93-a058-b62b038bd82a" containerName="manager" Jul 14 18:41:45.293025 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.293007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.295730 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.295709 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:41:45.295910 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.295729 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:41:45.295910 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.295748 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-r57vd\"" Jul 14 18:41:45.307599 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.307577 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl"] Jul 14 18:41:45.402273 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.402242 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjdz7\" (UniqueName: \"kubernetes.io/projected/538ed20c-81e1-415e-a6f9-6540e81adc1e-kube-api-access-tjdz7\") pod \"kuadrant-operator-controller-manager-86878c769f-hvndl\" (UID: \"538ed20c-81e1-415e-a6f9-6540e81adc1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.402439 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.402289 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/538ed20c-81e1-415e-a6f9-6540e81adc1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hvndl\" (UID: \"538ed20c-81e1-415e-a6f9-6540e81adc1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.503510 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.503479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjdz7\" (UniqueName: \"kubernetes.io/projected/538ed20c-81e1-415e-a6f9-6540e81adc1e-kube-api-access-tjdz7\") pod \"kuadrant-operator-controller-manager-86878c769f-hvndl\" (UID: \"538ed20c-81e1-415e-a6f9-6540e81adc1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.503670 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.503524 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/538ed20c-81e1-415e-a6f9-6540e81adc1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hvndl\" (UID: \"538ed20c-81e1-415e-a6f9-6540e81adc1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.503928 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.503911 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/538ed20c-81e1-415e-a6f9-6540e81adc1e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hvndl\" (UID: \"538ed20c-81e1-415e-a6f9-6540e81adc1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.516193 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.516167 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjdz7\" (UniqueName: \"kubernetes.io/projected/538ed20c-81e1-415e-a6f9-6540e81adc1e-kube-api-access-tjdz7\") pod \"kuadrant-operator-controller-manager-86878c769f-hvndl\" (UID: \"538ed20c-81e1-415e-a6f9-6540e81adc1e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.602571 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.602497 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:45.744103 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:45.744071 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl"] Jul 14 18:41:45.747902 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:41:45.747874 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod538ed20c_81e1_415e_a6f9_6540e81adc1e.slice/crio-e82a66ae90cdff5a3f6baf19b4f88931bce48607a653633d0021779ca9bbfa55 WatchSource:0}: Error finding container e82a66ae90cdff5a3f6baf19b4f88931bce48607a653633d0021779ca9bbfa55: Status 404 returned error can't find the container with id e82a66ae90cdff5a3f6baf19b4f88931bce48607a653633d0021779ca9bbfa55 Jul 14 18:41:46.565257 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:46.565216 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" event={"ID":"538ed20c-81e1-415e-a6f9-6540e81adc1e","Type":"ContainerStarted","Data":"0fce4bdac47bb0f34179d60ba981cb87629e386678ad660550f429d572216a97"} Jul 14 18:41:46.565615 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:46.565312 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" event={"ID":"538ed20c-81e1-415e-a6f9-6540e81adc1e","Type":"ContainerStarted","Data":"e82a66ae90cdff5a3f6baf19b4f88931bce48607a653633d0021779ca9bbfa55"} Jul 14 18:41:46.565615 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:46.565346 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:41:46.588793 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:46.588727 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" podStartSLOduration=1.58871249 podStartE2EDuration="1.58871249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:41:46.586693295 +0000 UTC m=+1197.311833378" watchObservedRunningTime="2026-07-14 18:41:46.58871249 +0000 UTC m=+1197.313852552" Jul 14 18:41:49.672426 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:49.672394 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:41:49.672912 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:49.672608 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:41:57.572178 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:41:57.572149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hvndl" Jul 14 18:46:49.696212 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:46:49.696181 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:46:49.698984 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:46:49.696870 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:48:02.083051 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.083024 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95"] Jul 14 18:48:02.086521 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.086499 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.088962 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.088941 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bfd487d2-openshift-default-dockercfg-sq962\"" Jul 14 18:48:02.118599 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.118574 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95"] Jul 14 18:48:02.134862 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.134834 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b48803-edbd-4c9e-9ce1-c6a914913802-istiod-ca-cert\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.134954 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.134879 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-socket\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.134954 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.134937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-certs\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.135029 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.134991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-token\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.135065 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.135025 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-data\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.135065 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.135048 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-podinfo\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.135129 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.135066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szn67\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-kube-api-access-szn67\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.135129 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.135086 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-credential-socket\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.135210 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.135194 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-envoy\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236290 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-envoy\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236454 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b48803-edbd-4c9e-9ce1-c6a914913802-istiod-ca-cert\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236454 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236430 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-socket\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236472 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-certs\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236523 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-token\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236577 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236561 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-data\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236731 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236604 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-podinfo\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236731 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236635 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szn67\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-kube-api-access-szn67\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236731 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236665 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-credential-socket\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236912 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-socket\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.236912 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236857 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-certs\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.237032 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.236996 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-data\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.237123 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.237099 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b48803-edbd-4c9e-9ce1-c6a914913802-istiod-ca-cert\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.237291 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.237270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-credential-socket\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.238821 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.238798 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-envoy\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.239222 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.239206 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-podinfo\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.246086 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.246061 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-token\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.246305 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.246288 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szn67\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-kube-api-access-szn67\") pod \"e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.397579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.397501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:02.534163 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.534138 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95"] Jul 14 18:48:02.538028 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:48:02.537996 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b48803_edbd_4c9e_9ce1_c6a914913802.slice/crio-9ac421bde515b8f54dce21bdbc3d19723cf529a982cfab7e065328e7f35ed9e1 WatchSource:0}: Error finding container 9ac421bde515b8f54dce21bdbc3d19723cf529a982cfab7e065328e7f35ed9e1: Status 404 returned error can't find the container with id 9ac421bde515b8f54dce21bdbc3d19723cf529a982cfab7e065328e7f35ed9e1 Jul 14 18:48:02.540453 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:02.540437 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:48:03.102010 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:03.101969 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" event={"ID":"06b48803-edbd-4c9e-9ce1-c6a914913802","Type":"ContainerStarted","Data":"9ac421bde515b8f54dce21bdbc3d19723cf529a982cfab7e065328e7f35ed9e1"} Jul 14 18:48:21.393208 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:21.393168 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:48:21.393520 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:21.393241 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:48:21.393520 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:21.393270 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:48:22.184063 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:22.184026 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" event={"ID":"06b48803-edbd-4c9e-9ce1-c6a914913802","Type":"ContainerStarted","Data":"286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f"} Jul 14 18:48:22.207478 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:22.207427 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podStartSLOduration=1.355127824 podStartE2EDuration="20.207411717s" totalImagesPullingTime="18.852283893s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:48:22.206040155 +0000 UTC m=+1592.931180221" watchObservedRunningTime="2026-07-14 18:48:22.207411717 +0000 UTC m=+1592.932551777" Jul 14 18:48:22.398079 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:22.398042 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:22.399552 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:22.399521 2580 patch_prober.go:28] interesting pod/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 14 18:48:22.399672 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:22.399581 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 14 18:48:23.398256 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:23.398218 2580 patch_prober.go:28] interesting pod/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 14 18:48:23.398632 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:23.398274 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 14 18:48:24.398723 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:24.398688 2580 patch_prober.go:28] interesting pod/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 14 18:48:24.399120 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:24.398778 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 14 18:48:25.398805 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:25.398742 2580 patch_prober.go:28] interesting pod/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 14 18:48:25.399176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:25.398823 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 14 18:48:26.397966 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:26.397929 2580 patch_prober.go:28] interesting pod/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" start-of-body= Jul 14 18:48:26.398145 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:26.397984 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.25:15021/healthz/ready\": dial tcp 10.134.0.25:15021: connect: connection refused" Jul 14 18:48:27.402516 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:27.402484 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:27.402966 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:27.402793 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:27.403507 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:27.403489 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:48:58.452363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.452278 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h"] Jul 14 18:48:58.455716 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.455695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:48:58.457999 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.457971 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:48:58.458106 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.458065 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:48:58.462952 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.462611 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h"] Jul 14 18:48:58.515485 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.515459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0229717e-93da-418d-96bd-d653824a02f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h\" (UID: \"0229717e-93da-418d-96bd-d653824a02f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:48:58.616490 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.616456 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0229717e-93da-418d-96bd-d653824a02f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h\" (UID: \"0229717e-93da-418d-96bd-d653824a02f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:48:58.617052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.617033 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0229717e-93da-418d-96bd-d653824a02f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h\" (UID: \"0229717e-93da-418d-96bd-d653824a02f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:48:58.766208 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.766186 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:48:58.896431 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:58.896403 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h"] Jul 14 18:48:58.898523 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:48:58.898496 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0229717e_93da_418d_96bd_d653824a02f2.slice/crio-7fe05da0731025adf572158ecc0f0a6855d8934150f2d7f39acee0a34f3ccb44 WatchSource:0}: Error finding container 7fe05da0731025adf572158ecc0f0a6855d8934150f2d7f39acee0a34f3ccb44: Status 404 returned error can't find the container with id 7fe05da0731025adf572158ecc0f0a6855d8934150f2d7f39acee0a34f3ccb44 Jul 14 18:48:59.334988 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:48:59.334954 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" event={"ID":"0229717e-93da-418d-96bd-d653824a02f2","Type":"ContainerStarted","Data":"7fe05da0731025adf572158ecc0f0a6855d8934150f2d7f39acee0a34f3ccb44"} Jul 14 18:49:01.345862 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:01.345829 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" event={"ID":"0229717e-93da-418d-96bd-d653824a02f2","Type":"ContainerStarted","Data":"cf6f71902fbdba2618ffe3f4523ebba4212bba2e17eaf62b4c6d823ce9281681"} Jul 14 18:49:01.382180 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:01.382134 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" podStartSLOduration=1.309260286 podStartE2EDuration="3.38211916s" totalImagesPullingTime="2.072858874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:49:01.381429106 +0000 UTC m=+1632.106569166" watchObservedRunningTime="2026-07-14 18:49:01.38211916 +0000 UTC m=+1632.107259221" Jul 14 18:49:02.350847 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:02.350816 2580 generic.go:356] "Generic (PLEG): container finished" podID="0229717e-93da-418d-96bd-d653824a02f2" containerID="cf6f71902fbdba2618ffe3f4523ebba4212bba2e17eaf62b4c6d823ce9281681" exitCode=0 Jul 14 18:49:02.351234 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:02.350873 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" event={"ID":"0229717e-93da-418d-96bd-d653824a02f2","Type":"ContainerDied","Data":"cf6f71902fbdba2618ffe3f4523ebba4212bba2e17eaf62b4c6d823ce9281681"} Jul 14 18:49:03.484686 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:03.484664 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:49:03.560221 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:03.560187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0229717e-93da-418d-96bd-d653824a02f2-maas-api-service-ca\") pod \"0229717e-93da-418d-96bd-d653824a02f2\" (UID: \"0229717e-93da-418d-96bd-d653824a02f2\") " Jul 14 18:49:03.560557 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:03.560533 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0229717e-93da-418d-96bd-d653824a02f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0229717e-93da-418d-96bd-d653824a02f2" (UID: "0229717e-93da-418d-96bd-d653824a02f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:49:03.661524 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:03.661452 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0229717e-93da-418d-96bd-d653824a02f2-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:04.361225 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:04.361180 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" event={"ID":"0229717e-93da-418d-96bd-d653824a02f2","Type":"ContainerDied","Data":"7fe05da0731025adf572158ecc0f0a6855d8934150f2d7f39acee0a34f3ccb44"} Jul 14 18:49:04.361225 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:04.361229 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fe05da0731025adf572158ecc0f0a6855d8934150f2d7f39acee0a34f3ccb44" Jul 14 18:49:04.361225 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:04.361196 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h" Jul 14 18:49:23.714262 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:23.714221 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95"] Jul 14 18:49:23.714731 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:23.714499 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" containerID="cri-o://286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f" gracePeriod=30 Jul 14 18:49:24.303918 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.303879 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl"] Jul 14 18:49:24.304207 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.304195 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0229717e-93da-418d-96bd-d653824a02f2" containerName="revoke-keys" Jul 14 18:49:24.304260 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.304209 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="0229717e-93da-418d-96bd-d653824a02f2" containerName="revoke-keys" Jul 14 18:49:24.304296 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.304264 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="0229717e-93da-418d-96bd-d653824a02f2" containerName="revoke-keys" Jul 14 18:49:24.310341 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.310322 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.313249 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.313226 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-541ed9bd-gw-openshift-default-dockercfg-fjb9s\"" Jul 14 18:49:24.322273 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.322248 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl"] Jul 14 18:49:24.433983 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.433953 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-envoy\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.433987 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-socket\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434005 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-credential-socket\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-token\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434066 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lwgd\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-kube-api-access-7lwgd\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-certs\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434157 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434109 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-data\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434346 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434202 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-podinfo\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.434346 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.434249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce1af5c-3f61-4a1c-8074-02475035d9f4-istiod-ca-cert\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.534737 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-token\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.534938 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534746 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lwgd\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-kube-api-access-7lwgd\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.534938 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-certs\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.534938 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-data\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.534938 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-podinfo\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.534938 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534903 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce1af5c-3f61-4a1c-8074-02475035d9f4-istiod-ca-cert\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535197 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.534997 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-envoy\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535197 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.535027 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-socket\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535197 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.535054 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-credential-socket\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535345 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.535254 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-certs\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535345 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.535303 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-data\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535462 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.535356 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-credential-socket\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.535509 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.535474 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-socket\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.536277 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.536252 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce1af5c-3f61-4a1c-8074-02475035d9f4-istiod-ca-cert\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.537464 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.537428 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-envoy\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.537626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.537609 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-podinfo\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.544380 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.544357 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-token\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.544538 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.544513 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lwgd\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-kube-api-access-7lwgd\") pod \"e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.620676 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.620598 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:24.789947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.789591 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl"] Jul 14 18:49:24.792699 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:49:24.792667 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce1af5c_3f61_4a1c_8074_02475035d9f4.slice/crio-92be5e505c292d11223b2071282590e96eee94dda890041bb0e5723d6d097ca3 WatchSource:0}: Error finding container 92be5e505c292d11223b2071282590e96eee94dda890041bb0e5723d6d097ca3: Status 404 returned error can't find the container with id 92be5e505c292d11223b2071282590e96eee94dda890041bb0e5723d6d097ca3 Jul 14 18:49:24.795366 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.795324 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:49:24.795455 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.795438 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:49:24.795512 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:24.795498 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:49:25.448723 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:25.448686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" event={"ID":"fce1af5c-3f61-4a1c-8074-02475035d9f4","Type":"ContainerStarted","Data":"e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a"} Jul 14 18:49:25.448909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:25.448796 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" event={"ID":"fce1af5c-3f61-4a1c-8074-02475035d9f4","Type":"ContainerStarted","Data":"92be5e505c292d11223b2071282590e96eee94dda890041bb0e5723d6d097ca3"} Jul 14 18:49:25.473020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:25.472960 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podStartSLOduration=1.472940859 podStartE2EDuration="1.472940859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:49:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:49:25.471976885 +0000 UTC m=+1656.197116946" watchObservedRunningTime="2026-07-14 18:49:25.472940859 +0000 UTC m=+1656.198080921" Jul 14 18:49:25.621672 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:25.621631 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:25.626561 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:25.626534 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:26.452620 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:26.452581 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:26.453414 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:26.453395 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:27.405469 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:27.405431 2580 patch_prober.go:28] interesting pod/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:49:27.405633 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:27.405497 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:49:28.999097 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:28.999042 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:49:29.075673 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075641 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-socket\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.075874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075688 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szn67\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-kube-api-access-szn67\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.075874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075746 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b48803-edbd-4c9e-9ce1-c6a914913802-istiod-ca-cert\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.075874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075809 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-podinfo\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.075874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075850 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-token\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.075874 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075876 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-data\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.076138 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075955 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-envoy\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.076138 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.075952 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:49:29.076138 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076005 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-credential-socket\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.076138 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076050 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-certs\") pod \"06b48803-edbd-4c9e-9ce1-c6a914913802\" (UID: \"06b48803-edbd-4c9e-9ce1-c6a914913802\") " Jul 14 18:49:29.076346 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076159 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06b48803-edbd-4c9e-9ce1-c6a914913802-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:49:29.076412 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076374 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06b48803-edbd-4c9e-9ce1-c6a914913802-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.076412 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076395 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.076518 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076402 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:49:29.076647 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.076612 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:49:29.077245 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.077194 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-data" (OuterVolumeSpecName: "istio-data") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:49:29.078933 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.078880 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-token" (OuterVolumeSpecName: "istio-token") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:49:29.079055 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.079034 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:49:29.079120 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.079058 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:49:29.079432 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.079403 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-kube-api-access-szn67" (OuterVolumeSpecName: "kube-api-access-szn67") pod "06b48803-edbd-4c9e-9ce1-c6a914913802" (UID: "06b48803-edbd-4c9e-9ce1-c6a914913802"). InnerVolumeSpecName "kube-api-access-szn67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:49:29.177361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177326 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.177361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177356 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.177361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177365 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.177361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177374 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.177622 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177383 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06b48803-edbd-4c9e-9ce1-c6a914913802-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.177622 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177393 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szn67\" (UniqueName: \"kubernetes.io/projected/06b48803-edbd-4c9e-9ce1-c6a914913802-kube-api-access-szn67\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.177622 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.177401 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06b48803-edbd-4c9e-9ce1-c6a914913802-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:29.469043 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.469009 2580 generic.go:356] "Generic (PLEG): container finished" podID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerID="286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f" exitCode=0 Jul 14 18:49:29.469213 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.469052 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" Jul 14 18:49:29.469213 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.469089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" event={"ID":"06b48803-edbd-4c9e-9ce1-c6a914913802","Type":"ContainerDied","Data":"286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f"} Jul 14 18:49:29.469213 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.469149 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95" event={"ID":"06b48803-edbd-4c9e-9ce1-c6a914913802","Type":"ContainerDied","Data":"9ac421bde515b8f54dce21bdbc3d19723cf529a982cfab7e065328e7f35ed9e1"} Jul 14 18:49:29.469213 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.469179 2580 scope.go:117] "RemoveContainer" containerID="286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f" Jul 14 18:49:29.480891 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.479568 2580 scope.go:117] "RemoveContainer" containerID="286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f" Jul 14 18:49:29.480891 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:49:29.480042 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f\": container with ID starting with 286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f not found: ID does not exist" containerID="286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f" Jul 14 18:49:29.480891 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.480081 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f"} err="failed to get container status \"286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f\": rpc error: code = NotFound desc = could not find container \"286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f\": container with ID starting with 286eab0b2ee4feaaee03b3b1a663696657c7df2e90c85e191f9c47f9d5a1907f not found: ID does not exist" Jul 14 18:49:29.496002 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.495971 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95"] Jul 14 18:49:29.500531 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.500510 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bfd487d2-openshift-default-fb7c5d66f-c8h95"] Jul 14 18:49:29.783922 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:29.783887 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" path="/var/lib/kubelet/pods/06b48803-edbd-4c9e-9ce1-c6a914913802/volumes" Jul 14 18:49:30.150787 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.150270 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg"] Jul 14 18:49:30.151250 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.150850 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" Jul 14 18:49:30.151250 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.150870 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" Jul 14 18:49:30.151250 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.150977 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="06b48803-edbd-4c9e-9ce1-c6a914913802" containerName="istio-proxy" Jul 14 18:49:30.155660 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.155636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:30.158514 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.158299 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:49:30.158514 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.158410 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:49:30.158514 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.158413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg"] Jul 14 18:49:30.287447 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.287407 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96bb5fe-887f-441d-b039-5e8ca02492f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg\" (UID: \"b96bb5fe-887f-441d-b039-5e8ca02492f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:30.388656 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.388567 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96bb5fe-887f-441d-b039-5e8ca02492f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg\" (UID: \"b96bb5fe-887f-441d-b039-5e8ca02492f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:30.389237 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.389217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96bb5fe-887f-441d-b039-5e8ca02492f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg\" (UID: \"b96bb5fe-887f-441d-b039-5e8ca02492f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:30.468079 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.468046 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:30.641837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:30.639739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg"] Jul 14 18:49:30.653194 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:49:30.653168 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb96bb5fe_887f_441d_b039_5e8ca02492f5.slice/crio-5cb46016397d776b1964d6a9d989825f7b8f370c305d9c761284da5f9485a0f5 WatchSource:0}: Error finding container 5cb46016397d776b1964d6a9d989825f7b8f370c305d9c761284da5f9485a0f5: Status 404 returned error can't find the container with id 5cb46016397d776b1964d6a9d989825f7b8f370c305d9c761284da5f9485a0f5 Jul 14 18:49:31.481940 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:31.481846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" event={"ID":"b96bb5fe-887f-441d-b039-5e8ca02492f5","Type":"ContainerStarted","Data":"95843656b8f727b2bf77b5cdab2017ab1d104cf7e12d2c741277e3d6b2030a9a"} Jul 14 18:49:31.481940 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:31.481909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" event={"ID":"b96bb5fe-887f-441d-b039-5e8ca02492f5","Type":"ContainerStarted","Data":"5cb46016397d776b1964d6a9d989825f7b8f370c305d9c761284da5f9485a0f5"} Jul 14 18:49:31.500159 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:31.500098 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" podStartSLOduration=1.500078019 podStartE2EDuration="1.500078019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:49:31.497517451 +0000 UTC m=+1662.222657514" watchObservedRunningTime="2026-07-14 18:49:31.500078019 +0000 UTC m=+1662.225218081" Jul 14 18:49:32.488474 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:32.488436 2580 generic.go:356] "Generic (PLEG): container finished" podID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerID="95843656b8f727b2bf77b5cdab2017ab1d104cf7e12d2c741277e3d6b2030a9a" exitCode=7 Jul 14 18:49:32.488910 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:32.488521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" event={"ID":"b96bb5fe-887f-441d-b039-5e8ca02492f5","Type":"ContainerDied","Data":"95843656b8f727b2bf77b5cdab2017ab1d104cf7e12d2c741277e3d6b2030a9a"} Jul 14 18:49:32.488910 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:32.488738 2580 scope.go:117] "RemoveContainer" containerID="95843656b8f727b2bf77b5cdab2017ab1d104cf7e12d2c741277e3d6b2030a9a" Jul 14 18:49:33.495305 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:33.495214 2580 generic.go:356] "Generic (PLEG): container finished" podID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerID="acfff41faf49a3b25c7820783467a06e2faa49a07b464515f85ebc44f458f4da" exitCode=7 Jul 14 18:49:33.495305 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:33.495267 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" event={"ID":"b96bb5fe-887f-441d-b039-5e8ca02492f5","Type":"ContainerDied","Data":"acfff41faf49a3b25c7820783467a06e2faa49a07b464515f85ebc44f458f4da"} Jul 14 18:49:33.495825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:33.495366 2580 scope.go:117] "RemoveContainer" containerID="95843656b8f727b2bf77b5cdab2017ab1d104cf7e12d2c741277e3d6b2030a9a" Jul 14 18:49:33.495825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:33.495544 2580 scope.go:117] "RemoveContainer" containerID="acfff41faf49a3b25c7820783467a06e2faa49a07b464515f85ebc44f458f4da" Jul 14 18:49:33.495825 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:49:33.495805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg_odh-ai-gateway-infra(b96bb5fe-887f-441d-b039-5e8ca02492f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" Jul 14 18:49:34.501918 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:34.501889 2580 scope.go:117] "RemoveContainer" containerID="acfff41faf49a3b25c7820783467a06e2faa49a07b464515f85ebc44f458f4da" Jul 14 18:49:34.502361 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:49:34.502109 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg_odh-ai-gateway-infra(b96bb5fe-887f-441d-b039-5e8ca02492f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" Jul 14 18:49:36.511307 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:36.511272 2580 generic.go:356] "Generic (PLEG): container finished" podID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerID="e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a" exitCode=0 Jul 14 18:49:36.511714 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:36.511354 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" event={"ID":"fce1af5c-3f61-4a1c-8074-02475035d9f4","Type":"ContainerDied","Data":"e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a"} Jul 14 18:49:36.511832 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:36.511816 2580 scope.go:117] "RemoveContainer" containerID="e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a" Jul 14 18:49:36.512782 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:36.512733 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:49:36.512887 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:36.512843 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:49:36.512951 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:36.512905 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:49:37.517749 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:37.517710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" event={"ID":"fce1af5c-3f61-4a1c-8074-02475035d9f4","Type":"ContainerStarted","Data":"14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47"} Jul 14 18:49:37.621546 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:37.621494 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:37.623039 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:37.622655 2580 patch_prober.go:28] interesting pod/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 14 18:49:37.623039 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:37.622722 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 14 18:49:38.621251 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:38.621215 2580 patch_prober.go:28] interesting pod/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 14 18:49:38.621638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:38.621269 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 14 18:49:39.620925 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:39.620893 2580 patch_prober.go:28] interesting pod/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 14 18:49:39.621092 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:39.620946 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 14 18:49:40.621725 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:40.621692 2580 patch_prober.go:28] interesting pod/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 14 18:49:40.622223 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:40.621751 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 14 18:49:41.453321 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:41.453281 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:41.621084 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:41.621051 2580 patch_prober.go:28] interesting pod/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" start-of-body= Jul 14 18:49:41.621252 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:41.621103 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.27:15021/healthz/ready\": dial tcp 10.134.0.27:15021: connect: connection refused" Jul 14 18:49:42.625388 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:42.625358 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:42.626403 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:42.626383 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:49:46.778300 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:46.778268 2580 scope.go:117] "RemoveContainer" containerID="acfff41faf49a3b25c7820783467a06e2faa49a07b464515f85ebc44f458f4da" Jul 14 18:49:47.580353 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:47.580319 2580 generic.go:356] "Generic (PLEG): container finished" podID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerID="536dc610888af0c965c4d544882262ad1914441781a6efc02985e32aa41a5b33" exitCode=0 Jul 14 18:49:47.580540 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:47.580378 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" event={"ID":"b96bb5fe-887f-441d-b039-5e8ca02492f5","Type":"ContainerDied","Data":"536dc610888af0c965c4d544882262ad1914441781a6efc02985e32aa41a5b33"} Jul 14 18:49:47.580540 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:47.580431 2580 scope.go:117] "RemoveContainer" containerID="acfff41faf49a3b25c7820783467a06e2faa49a07b464515f85ebc44f458f4da" Jul 14 18:49:48.614554 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:48.614519 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg"] Jul 14 18:49:48.712136 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:48.712113 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:48.756495 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:48.756465 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96bb5fe-887f-441d-b039-5e8ca02492f5-maas-api-service-ca\") pod \"b96bb5fe-887f-441d-b039-5e8ca02492f5\" (UID: \"b96bb5fe-887f-441d-b039-5e8ca02492f5\") " Jul 14 18:49:48.756803 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:48.756781 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b96bb5fe-887f-441d-b039-5e8ca02492f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b96bb5fe-887f-441d-b039-5e8ca02492f5" (UID: "b96bb5fe-887f-441d-b039-5e8ca02492f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:49:48.857988 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:48.857914 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b96bb5fe-887f-441d-b039-5e8ca02492f5-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:49:49.590698 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:49.590676 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" Jul 14 18:49:49.590855 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:49.590675 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg" event={"ID":"b96bb5fe-887f-441d-b039-5e8ca02492f5","Type":"ContainerDied","Data":"5cb46016397d776b1964d6a9d989825f7b8f370c305d9c761284da5f9485a0f5"} Jul 14 18:49:49.590855 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:49.590821 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb46016397d776b1964d6a9d989825f7b8f370c305d9c761284da5f9485a0f5" Jul 14 18:49:49.628723 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:49.628696 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg"] Jul 14 18:49:49.639022 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:49.638999 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-csgcg"] Jul 14 18:49:49.781719 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:49:49.781681 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" path="/var/lib/kubelet/pods/b96bb5fe-887f-441d-b039-5e8ca02492f5/volumes" Jul 14 18:50:20.204603 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.204562 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5"] Jul 14 18:50:20.205158 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205029 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205158 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205049 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205158 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205067 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205158 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205075 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205158 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205094 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205158 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205103 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205446 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205184 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205446 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205194 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.205446 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.205205 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="b96bb5fe-887f-441d-b039-5e8ca02492f5" containerName="revoke-keys" Jul 14 18:50:20.208076 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.208054 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:20.210684 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.210650 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:50:20.210809 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.210788 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:50:20.215806 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.215782 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5"] Jul 14 18:50:20.314152 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.314117 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318f94a0-46fe-455a-97ff-84d9951ec42f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5\" (UID: \"318f94a0-46fe-455a-97ff-84d9951ec42f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:20.415459 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.415419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318f94a0-46fe-455a-97ff-84d9951ec42f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5\" (UID: \"318f94a0-46fe-455a-97ff-84d9951ec42f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:20.416101 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.416076 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318f94a0-46fe-455a-97ff-84d9951ec42f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5\" (UID: \"318f94a0-46fe-455a-97ff-84d9951ec42f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:20.518854 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.518824 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:20.648072 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.648042 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5"] Jul 14 18:50:20.651489 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:50:20.651442 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318f94a0_46fe_455a_97ff_84d9951ec42f.slice/crio-de6e152563f0b2adadf66b1262b0378a750fac3baa55c4b6952265a46c62f71d WatchSource:0}: Error finding container de6e152563f0b2adadf66b1262b0378a750fac3baa55c4b6952265a46c62f71d: Status 404 returned error can't find the container with id de6e152563f0b2adadf66b1262b0378a750fac3baa55c4b6952265a46c62f71d Jul 14 18:50:20.727427 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:20.727402 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" event={"ID":"318f94a0-46fe-455a-97ff-84d9951ec42f","Type":"ContainerStarted","Data":"de6e152563f0b2adadf66b1262b0378a750fac3baa55c4b6952265a46c62f71d"} Jul 14 18:50:21.732664 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:21.732627 2580 generic.go:356] "Generic (PLEG): container finished" podID="318f94a0-46fe-455a-97ff-84d9951ec42f" containerID="64052e6fbe276d3c09f21f3fb1c70774ee0d25434d1ea99cc1f0b45b6371c563" exitCode=0 Jul 14 18:50:21.733069 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:21.732702 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" event={"ID":"318f94a0-46fe-455a-97ff-84d9951ec42f","Type":"ContainerDied","Data":"64052e6fbe276d3c09f21f3fb1c70774ee0d25434d1ea99cc1f0b45b6371c563"} Jul 14 18:50:22.864429 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:22.864406 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:22.935330 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:22.935295 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318f94a0-46fe-455a-97ff-84d9951ec42f-maas-api-service-ca\") pod \"318f94a0-46fe-455a-97ff-84d9951ec42f\" (UID: \"318f94a0-46fe-455a-97ff-84d9951ec42f\") " Jul 14 18:50:22.935675 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:22.935651 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318f94a0-46fe-455a-97ff-84d9951ec42f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "318f94a0-46fe-455a-97ff-84d9951ec42f" (UID: "318f94a0-46fe-455a-97ff-84d9951ec42f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:50:23.036755 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:23.036729 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318f94a0-46fe-455a-97ff-84d9951ec42f-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:23.744443 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:23.744399 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" event={"ID":"318f94a0-46fe-455a-97ff-84d9951ec42f","Type":"ContainerDied","Data":"de6e152563f0b2adadf66b1262b0378a750fac3baa55c4b6952265a46c62f71d"} Jul 14 18:50:23.744443 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:23.744428 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5" Jul 14 18:50:23.744667 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:23.744458 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de6e152563f0b2adadf66b1262b0378a750fac3baa55c4b6952265a46c62f71d" Jul 14 18:50:30.299798 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:30.299739 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl"] Jul 14 18:50:30.300237 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:30.300033 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" containerID="cri-o://14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47" gracePeriod=30 Jul 14 18:50:35.556222 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.556198 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:50:35.748793 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748738 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7lwgd\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-kube-api-access-7lwgd\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748806 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-envoy\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748827 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce1af5c-3f61-4a1c-8074-02475035d9f4-istiod-ca-cert\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748854 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-socket\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748883 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-data\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748900 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-podinfo\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-credential-socket\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.748949 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748935 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-certs\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.749275 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.748985 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-token\") pod \"fce1af5c-3f61-4a1c-8074-02475035d9f4\" (UID: \"fce1af5c-3f61-4a1c-8074-02475035d9f4\") " Jul 14 18:50:35.749275 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.749148 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:50:35.749275 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.749243 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.749419 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.749332 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:50:35.749620 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.749442 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:50:35.749620 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.749585 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fce1af5c-3f61-4a1c-8074-02475035d9f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:50:35.750254 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.750218 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:50:35.751207 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.751179 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:50:35.751207 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.751186 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:50:35.751319 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.751231 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-kube-api-access-7lwgd" (OuterVolumeSpecName: "kube-api-access-7lwgd") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "kube-api-access-7lwgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:50:35.751319 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.751256 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "fce1af5c-3f61-4a1c-8074-02475035d9f4" (UID: "fce1af5c-3f61-4a1c-8074-02475035d9f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:50:35.795792 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.795743 2580 generic.go:356] "Generic (PLEG): container finished" podID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerID="14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47" exitCode=0 Jul 14 18:50:35.795899 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.795828 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" Jul 14 18:50:35.795946 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.795826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" event={"ID":"fce1af5c-3f61-4a1c-8074-02475035d9f4","Type":"ContainerDied","Data":"14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47"} Jul 14 18:50:35.795946 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.795939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl" event={"ID":"fce1af5c-3f61-4a1c-8074-02475035d9f4","Type":"ContainerDied","Data":"92be5e505c292d11223b2071282590e96eee94dda890041bb0e5723d6d097ca3"} Jul 14 18:50:35.796008 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.795962 2580 scope.go:117] "RemoveContainer" containerID="14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47" Jul 14 18:50:35.804474 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.804450 2580 scope.go:117] "RemoveContainer" containerID="e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a" Jul 14 18:50:35.812134 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.812115 2580 scope.go:117] "RemoveContainer" containerID="14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47" Jul 14 18:50:35.812394 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:50:35.812375 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47\": container with ID starting with 14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47 not found: ID does not exist" containerID="14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47" Jul 14 18:50:35.812463 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.812402 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47"} err="failed to get container status \"14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47\": rpc error: code = NotFound desc = could not find container \"14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47\": container with ID starting with 14aadd12687e8b384506c9dba1bed6c97b1aca4a64745f9b248541206429eb47 not found: ID does not exist" Jul 14 18:50:35.812463 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.812421 2580 scope.go:117] "RemoveContainer" containerID="e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a" Jul 14 18:50:35.812660 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:50:35.812642 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a\": container with ID starting with e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a not found: ID does not exist" containerID="e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a" Jul 14 18:50:35.812779 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.812663 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a"} err="failed to get container status \"e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a\": rpc error: code = NotFound desc = could not find container \"e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a\": container with ID starting with e8db0bc76024dffa4ba59f6a8360fb500a59767f1fd4a97640dafc00947b882a not found: ID does not exist" Jul 14 18:50:35.815004 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.814981 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl"] Jul 14 18:50:35.820043 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.820022 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-541ed9bd-gw-openshift-default-5998d9cf95-lc8dl"] Jul 14 18:50:35.849621 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849602 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lwgd\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-kube-api-access-7lwgd\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849621 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849623 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849633 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fce1af5c-3f61-4a1c-8074-02475035d9f4-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849641 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849649 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849656 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849664 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fce1af5c-3f61-4a1c-8074-02475035d9f4-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:35.849732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:35.849671 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fce1af5c-3f61-4a1c-8074-02475035d9f4-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:50:37.357784 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.357738 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg"] Jul 14 18:50:37.358227 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358188 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" Jul 14 18:50:37.358227 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358205 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" Jul 14 18:50:37.358227 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358225 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318f94a0-46fe-455a-97ff-84d9951ec42f" containerName="revoke-keys" Jul 14 18:50:37.358384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358234 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="318f94a0-46fe-455a-97ff-84d9951ec42f" containerName="revoke-keys" Jul 14 18:50:37.358384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358321 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" Jul 14 18:50:37.358384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358337 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" Jul 14 18:50:37.358384 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.358346 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="318f94a0-46fe-455a-97ff-84d9951ec42f" containerName="revoke-keys" Jul 14 18:50:37.362918 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.362897 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.365516 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.365493 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-01bc759f-openshift-default-dockercfg-rx5kl\"" Jul 14 18:50:37.374878 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.374853 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg"] Jul 14 18:50:37.464471 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464438 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-data\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464677 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464500 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-podinfo\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464677 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-token\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464677 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-envoy\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464886 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464677 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-socket\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464886 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464802 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23660e6-d2fa-40ca-be99-b93bba8981ad-istiod-ca-cert\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464886 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm48z\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-kube-api-access-pm48z\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.464886 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464864 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-certs\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.465086 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.464888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-credential-socket\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566262 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-data\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566262 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-podinfo\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566278 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-token\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566311 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-envoy\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566357 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-socket\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23660e6-d2fa-40ca-be99-b93bba8981ad-istiod-ca-cert\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm48z\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-kube-api-access-pm48z\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566496 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566479 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-certs\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566840 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-credential-socket\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.566840 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566813 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-socket\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.567010 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-credential-socket\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.567010 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.566918 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-certs\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.567083 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.567039 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-data\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.567176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.567156 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23660e6-d2fa-40ca-be99-b93bba8981ad-istiod-ca-cert\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.569183 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.569161 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-envoy\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.569381 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.569363 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-podinfo\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.575373 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.575349 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-token\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.575938 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.575904 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm48z\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-kube-api-access-pm48z\") pod \"e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.674545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.674446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:37.783571 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.783537 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" path="/var/lib/kubelet/pods/fce1af5c-3f61-4a1c-8074-02475035d9f4/volumes" Jul 14 18:50:37.810028 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.810007 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg"] Jul 14 18:50:37.812147 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:50:37.812116 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23660e6_d2fa_40ca_be99_b93bba8981ad.slice/crio-e19bd9c1980bb6db3623615919fc453ba4841b339ac1da6cb880831062ed4ba6 WatchSource:0}: Error finding container e19bd9c1980bb6db3623615919fc453ba4841b339ac1da6cb880831062ed4ba6: Status 404 returned error can't find the container with id e19bd9c1980bb6db3623615919fc453ba4841b339ac1da6cb880831062ed4ba6 Jul 14 18:50:37.814069 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.814035 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:50:37.814176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.814098 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:50:37.814176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:37.814125 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:50:38.809855 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:38.809818 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" event={"ID":"e23660e6-d2fa-40ca-be99-b93bba8981ad","Type":"ContainerStarted","Data":"a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b"} Jul 14 18:50:38.810249 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:38.809875 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" event={"ID":"e23660e6-d2fa-40ca-be99-b93bba8981ad","Type":"ContainerStarted","Data":"e19bd9c1980bb6db3623615919fc453ba4841b339ac1da6cb880831062ed4ba6"} Jul 14 18:50:38.843023 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:38.842959 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" podStartSLOduration=1.842940103 podStartE2EDuration="1.842940103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:50:38.840919561 +0000 UTC m=+1729.566059629" watchObservedRunningTime="2026-07-14 18:50:38.842940103 +0000 UTC m=+1729.568080166" Jul 14 18:50:39.675361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:39.675333 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:39.679906 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:39.679881 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:39.813700 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:39.813667 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:39.814629 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:39.814613 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:43.789258 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.789228 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw"] Jul 14 18:50:43.789708 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.789525 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" Jul 14 18:50:43.789708 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.789535 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fce1af5c-3f61-4a1c-8074-02475035d9f4" containerName="istio-proxy" Jul 14 18:50:43.792162 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.792146 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:50:43.796885 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.796861 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:50:43.796992 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.796900 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:50:43.803311 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.803290 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw"] Jul 14 18:50:43.810165 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.810131 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed13d47-74de-4e6b-b736-bad12a97238d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw\" (UID: \"3ed13d47-74de-4e6b-b736-bad12a97238d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:50:43.911077 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.911045 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed13d47-74de-4e6b-b736-bad12a97238d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw\" (UID: \"3ed13d47-74de-4e6b-b736-bad12a97238d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:50:43.911626 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:43.911604 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed13d47-74de-4e6b-b736-bad12a97238d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw\" (UID: \"3ed13d47-74de-4e6b-b736-bad12a97238d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:50:44.102508 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:44.102418 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:50:44.231752 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:44.231721 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw"] Jul 14 18:50:44.232933 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:50:44.232908 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ed13d47_74de_4e6b_b736_bad12a97238d.slice/crio-f66545b78c646548c54a7176bab7edfd78c90d21c817ff7d050f137da1732cec WatchSource:0}: Error finding container f66545b78c646548c54a7176bab7edfd78c90d21c817ff7d050f137da1732cec: Status 404 returned error can't find the container with id f66545b78c646548c54a7176bab7edfd78c90d21c817ff7d050f137da1732cec Jul 14 18:50:44.845986 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:44.845946 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" event={"ID":"3ed13d47-74de-4e6b-b736-bad12a97238d","Type":"ContainerStarted","Data":"c01d4bfa70b1f75dcaf41f50eadfa9ebc9acff3de1c3da755a12f0047393ec9a"} Jul 14 18:50:44.846465 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:44.846036 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" event={"ID":"3ed13d47-74de-4e6b-b736-bad12a97238d","Type":"ContainerStarted","Data":"f66545b78c646548c54a7176bab7edfd78c90d21c817ff7d050f137da1732cec"} Jul 14 18:50:44.866481 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:44.866410 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" podStartSLOduration=1.8663893740000002 podStartE2EDuration="1.866389374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:50:44.864468884 +0000 UTC m=+1735.589608947" watchObservedRunningTime="2026-07-14 18:50:44.866389374 +0000 UTC m=+1735.591529438" Jul 14 18:50:45.852257 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:45.852224 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerID="c01d4bfa70b1f75dcaf41f50eadfa9ebc9acff3de1c3da755a12f0047393ec9a" exitCode=7 Jul 14 18:50:45.852671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:45.852301 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" event={"ID":"3ed13d47-74de-4e6b-b736-bad12a97238d","Type":"ContainerDied","Data":"c01d4bfa70b1f75dcaf41f50eadfa9ebc9acff3de1c3da755a12f0047393ec9a"} Jul 14 18:50:45.852671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:45.852483 2580 scope.go:117] "RemoveContainer" containerID="c01d4bfa70b1f75dcaf41f50eadfa9ebc9acff3de1c3da755a12f0047393ec9a" Jul 14 18:50:46.859894 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:46.859859 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerID="07b6498f03c22332917e2ea82375203be86d76652f6b01683a4b8da49bb5e3b2" exitCode=7 Jul 14 18:50:46.860328 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:46.859904 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" event={"ID":"3ed13d47-74de-4e6b-b736-bad12a97238d","Type":"ContainerDied","Data":"07b6498f03c22332917e2ea82375203be86d76652f6b01683a4b8da49bb5e3b2"} Jul 14 18:50:46.860328 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:46.859967 2580 scope.go:117] "RemoveContainer" containerID="c01d4bfa70b1f75dcaf41f50eadfa9ebc9acff3de1c3da755a12f0047393ec9a" Jul 14 18:50:46.860328 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:46.860164 2580 scope.go:117] "RemoveContainer" containerID="07b6498f03c22332917e2ea82375203be86d76652f6b01683a4b8da49bb5e3b2" Jul 14 18:50:46.860474 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:50:46.860450 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw_odh-ai-gateway-infra(3ed13d47-74de-4e6b-b736-bad12a97238d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" Jul 14 18:50:47.865179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:47.865150 2580 scope.go:117] "RemoveContainer" containerID="07b6498f03c22332917e2ea82375203be86d76652f6b01683a4b8da49bb5e3b2" Jul 14 18:50:47.865554 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:50:47.865327 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw_odh-ai-gateway-infra(3ed13d47-74de-4e6b-b736-bad12a97238d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" Jul 14 18:50:51.888900 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:51.888809 2580 generic.go:356] "Generic (PLEG): container finished" podID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerID="a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b" exitCode=0 Jul 14 18:50:51.889391 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:51.888889 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" event={"ID":"e23660e6-d2fa-40ca-be99-b93bba8981ad","Type":"ContainerDied","Data":"a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b"} Jul 14 18:50:51.889391 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:51.889386 2580 scope.go:117] "RemoveContainer" containerID="a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b" Jul 14 18:50:51.890406 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:51.890381 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:50:51.890467 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:51.890444 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:50:51.890504 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:51.890474 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:50:52.895445 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:52.895414 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" event={"ID":"e23660e6-d2fa-40ca-be99-b93bba8981ad","Type":"ContainerStarted","Data":"3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6"} Jul 14 18:50:53.674600 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:53.674573 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:53.675715 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:53.675689 2580 patch_prober.go:28] interesting pod/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 14 18:50:53.675868 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:53.675743 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 14 18:50:54.675166 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:54.675127 2580 patch_prober.go:28] interesting pod/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 14 18:50:54.675543 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:54.675180 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 14 18:50:54.814225 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:54.814191 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:55.675548 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:55.675511 2580 patch_prober.go:28] interesting pod/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 14 18:50:55.675954 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:55.675564 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 14 18:50:56.674976 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:56.674940 2580 patch_prober.go:28] interesting pod/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body= Jul 14 18:50:56.675132 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:56.674995 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" Jul 14 18:50:57.678914 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:57.678880 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:57.679999 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:57.679977 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:50:58.777595 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:58.777558 2580 scope.go:117] "RemoveContainer" containerID="07b6498f03c22332917e2ea82375203be86d76652f6b01683a4b8da49bb5e3b2" Jul 14 18:50:59.926176 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:59.926142 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerID="1006cd3587253377155c57de108853f754233a6ed13f0b0a2c7870c45f606984" exitCode=0 Jul 14 18:50:59.926614 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:59.926214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" event={"ID":"3ed13d47-74de-4e6b-b736-bad12a97238d","Type":"ContainerDied","Data":"1006cd3587253377155c57de108853f754233a6ed13f0b0a2c7870c45f606984"} Jul 14 18:50:59.926614 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:50:59.926274 2580 scope.go:117] "RemoveContainer" containerID="07b6498f03c22332917e2ea82375203be86d76652f6b01683a4b8da49bb5e3b2" Jul 14 18:51:00.964254 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:00.964223 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw"] Jul 14 18:51:01.060879 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.060850 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:51:01.153952 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.153915 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed13d47-74de-4e6b-b736-bad12a97238d-maas-api-service-ca\") pod \"3ed13d47-74de-4e6b-b736-bad12a97238d\" (UID: \"3ed13d47-74de-4e6b-b736-bad12a97238d\") " Jul 14 18:51:01.154289 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.154270 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed13d47-74de-4e6b-b736-bad12a97238d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3ed13d47-74de-4e6b-b736-bad12a97238d" (UID: "3ed13d47-74de-4e6b-b736-bad12a97238d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:51:01.255404 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.255318 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3ed13d47-74de-4e6b-b736-bad12a97238d-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:51:01.937403 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.937375 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" Jul 14 18:51:01.937403 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.937393 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw" event={"ID":"3ed13d47-74de-4e6b-b736-bad12a97238d","Type":"ContainerDied","Data":"f66545b78c646548c54a7176bab7edfd78c90d21c817ff7d050f137da1732cec"} Jul 14 18:51:01.937595 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.937435 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f66545b78c646548c54a7176bab7edfd78c90d21c817ff7d050f137da1732cec" Jul 14 18:51:01.965045 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.965010 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw"] Jul 14 18:51:01.969096 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:01.969073 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-5v4xw"] Jul 14 18:51:03.783448 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:03.783414 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" path="/var/lib/kubelet/pods/3ed13d47-74de-4e6b-b736-bad12a97238d/volumes" Jul 14 18:51:33.847127 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847092 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx"] Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847385 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847396 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847410 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847416 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847422 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847429 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847484 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847492 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.847579 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.847497 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ed13d47-74de-4e6b-b736-bad12a97238d" containerName="revoke-keys" Jul 14 18:51:33.850310 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.850294 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:33.852892 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.852859 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:51:33.853009 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.852941 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:51:33.858064 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.858044 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx"] Jul 14 18:51:33.946508 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:33.946471 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e61b2-536c-435f-858e-75fe1f9172e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx\" (UID: \"546e61b2-536c-435f-858e-75fe1f9172e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:34.047264 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:34.047225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e61b2-536c-435f-858e-75fe1f9172e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx\" (UID: \"546e61b2-536c-435f-858e-75fe1f9172e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:34.047908 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:34.047881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e61b2-536c-435f-858e-75fe1f9172e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx\" (UID: \"546e61b2-536c-435f-858e-75fe1f9172e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:34.161230 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:34.161149 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:34.287047 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:34.287024 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx"] Jul 14 18:51:34.289495 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:51:34.289469 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546e61b2_536c_435f_858e_75fe1f9172e3.slice/crio-006b015adeeabefe56ee3ee7a0d7a84716d6f83bc20f45ec0da28a16f559cca6 WatchSource:0}: Error finding container 006b015adeeabefe56ee3ee7a0d7a84716d6f83bc20f45ec0da28a16f559cca6: Status 404 returned error can't find the container with id 006b015adeeabefe56ee3ee7a0d7a84716d6f83bc20f45ec0da28a16f559cca6 Jul 14 18:51:35.084988 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:35.084955 2580 generic.go:356] "Generic (PLEG): container finished" podID="546e61b2-536c-435f-858e-75fe1f9172e3" containerID="2f037ea5d5d264457136e66479d171eb104b30c5116a0bee64fbda82973e4b1e" exitCode=0 Jul 14 18:51:35.085362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:35.085049 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" event={"ID":"546e61b2-536c-435f-858e-75fe1f9172e3","Type":"ContainerDied","Data":"2f037ea5d5d264457136e66479d171eb104b30c5116a0bee64fbda82973e4b1e"} Jul 14 18:51:35.085362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:35.085119 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" event={"ID":"546e61b2-536c-435f-858e-75fe1f9172e3","Type":"ContainerStarted","Data":"006b015adeeabefe56ee3ee7a0d7a84716d6f83bc20f45ec0da28a16f559cca6"} Jul 14 18:51:36.227783 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:36.227736 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:36.261479 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:36.261439 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e61b2-536c-435f-858e-75fe1f9172e3-maas-api-service-ca\") pod \"546e61b2-536c-435f-858e-75fe1f9172e3\" (UID: \"546e61b2-536c-435f-858e-75fe1f9172e3\") " Jul 14 18:51:36.261789 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:36.261744 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546e61b2-536c-435f-858e-75fe1f9172e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "546e61b2-536c-435f-858e-75fe1f9172e3" (UID: "546e61b2-536c-435f-858e-75fe1f9172e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:51:36.362026 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:36.361931 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e61b2-536c-435f-858e-75fe1f9172e3-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:51:37.095562 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:37.095533 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" Jul 14 18:51:37.095748 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:37.095574 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx" event={"ID":"546e61b2-536c-435f-858e-75fe1f9172e3","Type":"ContainerDied","Data":"006b015adeeabefe56ee3ee7a0d7a84716d6f83bc20f45ec0da28a16f559cca6"} Jul 14 18:51:37.095748 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:37.095622 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006b015adeeabefe56ee3ee7a0d7a84716d6f83bc20f45ec0da28a16f559cca6" Jul 14 18:51:49.720968 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:49.720936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:51:49.723339 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:51:49.723310 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:52:00.229330 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.229298 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg"] Jul 14 18:52:00.229802 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.229490 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" containerID="cri-o://3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6" gracePeriod=30 Jul 14 18:52:00.603971 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.603935 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5"] Jul 14 18:52:00.604423 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.604403 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546e61b2-536c-435f-858e-75fe1f9172e3" containerName="revoke-keys" Jul 14 18:52:00.604476 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.604429 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="546e61b2-536c-435f-858e-75fe1f9172e3" containerName="revoke-keys" Jul 14 18:52:00.604550 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.604538 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="546e61b2-536c-435f-858e-75fe1f9172e3" containerName="revoke-keys" Jul 14 18:52:00.607376 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.607350 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.610036 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.610008 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a8b422c9-openshift-default-dockercfg-42khc\"" Jul 14 18:52:00.616914 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.616891 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5"] Jul 14 18:52:00.671181 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-envoy\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671186 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-socket\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddb87\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-kube-api-access-ddb87\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671285 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-certs\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671323 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istiod-ca-cert\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671479 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671353 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-podinfo\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671479 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671431 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-credential-socket\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671479 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671462 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-token\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.671569 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.671496 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-data\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772446 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772408 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-token\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772446 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-data\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772681 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-envoy\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772681 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-socket\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772681 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772657 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddb87\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-kube-api-access-ddb87\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772690 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-certs\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istiod-ca-cert\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-podinfo\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772855 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-credential-socket\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.772877 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-data\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.773100 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.772999 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-socket\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.773168 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.773146 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-certs\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.773207 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.773176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-credential-socket\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.773413 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.773394 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istiod-ca-cert\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.775095 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.775075 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-envoy\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.775388 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.775369 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-podinfo\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.781598 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.781575 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-token\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.781909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.781890 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddb87\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-kube-api-access-ddb87\") pod \"e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:00.918736 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:00.918652 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:01.068647 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.068590 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5"] Jul 14 18:52:01.070903 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:52:01.070875 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f72fe17_86b3_4f67_bae4_c3cad56ba1b0.slice/crio-b7ca1d212094e186e0032bd356bd04ea6b5d341ce056bdd4f59b5a73feaf7a93 WatchSource:0}: Error finding container b7ca1d212094e186e0032bd356bd04ea6b5d341ce056bdd4f59b5a73feaf7a93: Status 404 returned error can't find the container with id b7ca1d212094e186e0032bd356bd04ea6b5d341ce056bdd4f59b5a73feaf7a93 Jul 14 18:52:01.073173 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.073141 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:52:01.073257 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.073204 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:52:01.073257 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.073234 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:52:01.196831 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.196716 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" event={"ID":"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0","Type":"ContainerStarted","Data":"83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812"} Jul 14 18:52:01.196831 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.196810 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" event={"ID":"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0","Type":"ContainerStarted","Data":"b7ca1d212094e186e0032bd356bd04ea6b5d341ce056bdd4f59b5a73feaf7a93"} Jul 14 18:52:01.223354 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.223294 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podStartSLOduration=1.223272075 podStartE2EDuration="1.223272075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:52:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:52:01.220445357 +0000 UTC m=+1811.945585419" watchObservedRunningTime="2026-07-14 18:52:01.223272075 +0000 UTC m=+1811.948412137" Jul 14 18:52:01.919004 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:01.918971 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:02.924660 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:02.924631 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:03.205936 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:03.205856 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:03.206729 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:03.206713 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:05.496883 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.496818 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:52:05.615837 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.615812 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-certs\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.615950 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.615876 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-socket\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.615950 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.615919 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-token\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.615950 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.615942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm48z\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-kube-api-access-pm48z\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.616102 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.615995 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-data\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.616102 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616019 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23660e6-d2fa-40ca-be99-b93bba8981ad-istiod-ca-cert\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.616102 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616047 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-credential-socket\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.616102 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616045 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:52:05.616102 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616082 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-podinfo\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.616334 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616146 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-envoy\") pod \"e23660e6-d2fa-40ca-be99-b93bba8981ad\" (UID: \"e23660e6-d2fa-40ca-be99-b93bba8981ad\") " Jul 14 18:52:05.616334 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616179 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:52:05.616403 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616386 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:52:05.616455 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616440 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.616515 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616460 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.616930 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.616891 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23660e6-d2fa-40ca-be99-b93bba8981ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:52:05.617358 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.617245 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:52:05.618379 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.618352 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:52:05.618479 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.618415 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-kube-api-access-pm48z" (OuterVolumeSpecName: "kube-api-access-pm48z") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "kube-api-access-pm48z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:52:05.618835 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.618817 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:52:05.618908 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.618868 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e23660e6-d2fa-40ca-be99-b93bba8981ad" (UID: "e23660e6-d2fa-40ca-be99-b93bba8981ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:52:05.717707 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717670 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.717707 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717706 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.717959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717720 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.717959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717732 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.717959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717743 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm48z\" (UniqueName: \"kubernetes.io/projected/e23660e6-d2fa-40ca-be99-b93bba8981ad-kube-api-access-pm48z\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.717959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717754 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e23660e6-d2fa-40ca-be99-b93bba8981ad-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:05.717959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:05.717789 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e23660e6-d2fa-40ca-be99-b93bba8981ad-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:06.220737 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.220643 2580 generic.go:356] "Generic (PLEG): container finished" podID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerID="3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6" exitCode=0 Jul 14 18:52:06.220737 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.220692 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" Jul 14 18:52:06.220943 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.220732 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" event={"ID":"e23660e6-d2fa-40ca-be99-b93bba8981ad","Type":"ContainerDied","Data":"3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6"} Jul 14 18:52:06.220943 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.220805 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg" event={"ID":"e23660e6-d2fa-40ca-be99-b93bba8981ad","Type":"ContainerDied","Data":"e19bd9c1980bb6db3623615919fc453ba4841b339ac1da6cb880831062ed4ba6"} Jul 14 18:52:06.220943 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.220829 2580 scope.go:117] "RemoveContainer" containerID="3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6" Jul 14 18:52:06.237029 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.237001 2580 scope.go:117] "RemoveContainer" containerID="a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b" Jul 14 18:52:06.243063 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.243037 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg"] Jul 14 18:52:06.246682 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.246656 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-01bc759f-openshift-default-74b89d4d5-gj5hg"] Jul 14 18:52:06.256640 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.256620 2580 scope.go:117] "RemoveContainer" containerID="3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6" Jul 14 18:52:06.257133 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:52:06.257110 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6\": container with ID starting with 3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6 not found: ID does not exist" containerID="3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6" Jul 14 18:52:06.257211 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.257142 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6"} err="failed to get container status \"3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6\": rpc error: code = NotFound desc = could not find container \"3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6\": container with ID starting with 3ddb70525ec57dc2563b7f724916b768e18bee5fdc87eb682d09c0a6532fffd6 not found: ID does not exist" Jul 14 18:52:06.257211 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.257161 2580 scope.go:117] "RemoveContainer" containerID="a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b" Jul 14 18:52:06.257494 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:52:06.257475 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b\": container with ID starting with a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b not found: ID does not exist" containerID="a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b" Jul 14 18:52:06.257557 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.257503 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b"} err="failed to get container status \"a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b\": rpc error: code = NotFound desc = could not find container \"a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b\": container with ID starting with a51cf754edb6b499779824593bfa9f51359b8571056911081150b9950522248b not found: ID does not exist" Jul 14 18:52:06.285125 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.285098 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl"] Jul 14 18:52:06.285407 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.285395 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" Jul 14 18:52:06.285450 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.285409 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" Jul 14 18:52:06.285483 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.285471 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" Jul 14 18:52:06.289865 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.289848 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:06.292301 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.292281 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:52:06.292401 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.292282 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:52:06.297364 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.297340 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl"] Jul 14 18:52:06.423223 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.423089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6ac338-c6fc-47f9-a15b-c12f377d1a8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl\" (UID: \"da6ac338-c6fc-47f9-a15b-c12f377d1a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:06.525493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.524599 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6ac338-c6fc-47f9-a15b-c12f377d1a8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl\" (UID: \"da6ac338-c6fc-47f9-a15b-c12f377d1a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:06.525493 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.525441 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6ac338-c6fc-47f9-a15b-c12f377d1a8f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl\" (UID: \"da6ac338-c6fc-47f9-a15b-c12f377d1a8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:06.601341 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.601311 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:06.808904 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:06.808794 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl"] Jul 14 18:52:06.812047 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:52:06.812019 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6ac338_c6fc_47f9_a15b_c12f377d1a8f.slice/crio-438dd873226eb2d9f79044ea9902df16f939c1371d279ed4c72b4173f555223c WatchSource:0}: Error finding container 438dd873226eb2d9f79044ea9902df16f939c1371d279ed4c72b4173f555223c: Status 404 returned error can't find the container with id 438dd873226eb2d9f79044ea9902df16f939c1371d279ed4c72b4173f555223c Jul 14 18:52:07.229498 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:07.229455 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerStarted","Data":"fadce81226c87237556b878b8e7fb970cc97bdedbc3a12a6d53b5bb44c48c763"} Jul 14 18:52:07.229695 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:07.229521 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerStarted","Data":"438dd873226eb2d9f79044ea9902df16f939c1371d279ed4c72b4173f555223c"} Jul 14 18:52:07.245594 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:07.245533 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" podStartSLOduration=1.245512298 podStartE2EDuration="1.245512298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:52:07.244384341 +0000 UTC m=+1817.969524403" watchObservedRunningTime="2026-07-14 18:52:07.245512298 +0000 UTC m=+1817.970652362" Jul 14 18:52:07.782361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:07.782327 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" path="/var/lib/kubelet/pods/e23660e6-d2fa-40ca-be99-b93bba8981ad/volumes" Jul 14 18:52:08.237805 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:08.237743 2580 generic.go:356] "Generic (PLEG): container finished" podID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerID="fadce81226c87237556b878b8e7fb970cc97bdedbc3a12a6d53b5bb44c48c763" exitCode=7 Jul 14 18:52:08.238020 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:08.237846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerDied","Data":"fadce81226c87237556b878b8e7fb970cc97bdedbc3a12a6d53b5bb44c48c763"} Jul 14 18:52:08.238185 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:08.238162 2580 scope.go:117] "RemoveContainer" containerID="fadce81226c87237556b878b8e7fb970cc97bdedbc3a12a6d53b5bb44c48c763" Jul 14 18:52:09.255308 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:09.255273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerStarted","Data":"a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91"} Jul 14 18:52:10.262880 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:10.262839 2580 generic.go:356] "Generic (PLEG): container finished" podID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerID="a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91" exitCode=7 Jul 14 18:52:10.263542 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:10.262895 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerDied","Data":"a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91"} Jul 14 18:52:10.263542 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:10.262970 2580 scope.go:117] "RemoveContainer" containerID="fadce81226c87237556b878b8e7fb970cc97bdedbc3a12a6d53b5bb44c48c763" Jul 14 18:52:10.263542 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:10.263189 2580 scope.go:117] "RemoveContainer" containerID="a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91" Jul 14 18:52:10.263542 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:52:10.263452 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl_odh-ai-gateway-infra(da6ac338-c6fc-47f9-a15b-c12f377d1a8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" Jul 14 18:52:11.270075 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:11.270048 2580 scope.go:117] "RemoveContainer" containerID="a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91" Jul 14 18:52:11.270427 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:52:11.270257 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl_odh-ai-gateway-infra(da6ac338-c6fc-47f9-a15b-c12f377d1a8f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" Jul 14 18:52:13.279632 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:13.279600 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerID="83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812" exitCode=0 Jul 14 18:52:13.280087 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:13.279676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" event={"ID":"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0","Type":"ContainerDied","Data":"83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812"} Jul 14 18:52:13.280087 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:13.280078 2580 scope.go:117] "RemoveContainer" containerID="83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812" Jul 14 18:52:13.281119 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:13.281094 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:52:13.281181 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:13.281159 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:52:13.281219 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:13.281188 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:52:14.285642 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:14.285608 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" event={"ID":"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0","Type":"ContainerStarted","Data":"114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e"} Jul 14 18:52:14.918988 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:14.918956 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:14.919885 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:14.919860 2580 patch_prober.go:28] interesting pod/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 14 18:52:14.920016 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:14.919918 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 14 18:52:15.920045 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:15.920009 2580 patch_prober.go:28] interesting pod/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 14 18:52:15.920477 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:15.920063 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 14 18:52:16.920108 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:16.920070 2580 patch_prober.go:28] interesting pod/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 14 18:52:16.920521 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:16.920123 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 14 18:52:17.919174 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:17.919131 2580 patch_prober.go:28] interesting pod/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Jul 14 18:52:17.919376 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:17.919191 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Jul 14 18:52:18.206936 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:18.206854 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:18.923560 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:18.923527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:18.924537 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:18.924518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:52:24.777917 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:24.777882 2580 scope.go:117] "RemoveContainer" containerID="a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91" Jul 14 18:52:25.333370 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:25.333281 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerStarted","Data":"d36a602c24ff1648e88791434e9b7cd189cc0bf5e4c4f2b33738264e72d20816"} Jul 14 18:52:26.339270 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:26.339242 2580 generic.go:356] "Generic (PLEG): container finished" podID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerID="d36a602c24ff1648e88791434e9b7cd189cc0bf5e4c4f2b33738264e72d20816" exitCode=0 Jul 14 18:52:26.339739 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:26.339319 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerDied","Data":"d36a602c24ff1648e88791434e9b7cd189cc0bf5e4c4f2b33738264e72d20816"} Jul 14 18:52:26.339739 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:26.339398 2580 scope.go:117] "RemoveContainer" containerID="a117be29839de5592840153943c71416a856fb320b705e081e00c81ea4c07e91" Jul 14 18:52:26.365773 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:26.365735 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl"] Jul 14 18:52:27.471828 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:27.471802 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:27.516989 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:27.516955 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6ac338-c6fc-47f9-a15b-c12f377d1a8f-maas-api-service-ca\") pod \"da6ac338-c6fc-47f9-a15b-c12f377d1a8f\" (UID: \"da6ac338-c6fc-47f9-a15b-c12f377d1a8f\") " Jul 14 18:52:27.517296 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:27.517273 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da6ac338-c6fc-47f9-a15b-c12f377d1a8f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da6ac338-c6fc-47f9-a15b-c12f377d1a8f" (UID: "da6ac338-c6fc-47f9-a15b-c12f377d1a8f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:52:27.618396 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:27.618312 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da6ac338-c6fc-47f9-a15b-c12f377d1a8f-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:52:28.349847 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:28.349808 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" Jul 14 18:52:28.349993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:28.349807 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl" event={"ID":"da6ac338-c6fc-47f9-a15b-c12f377d1a8f","Type":"ContainerDied","Data":"438dd873226eb2d9f79044ea9902df16f939c1371d279ed4c72b4173f555223c"} Jul 14 18:52:28.349993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:28.349975 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="438dd873226eb2d9f79044ea9902df16f939c1371d279ed4c72b4173f555223c" Jul 14 18:52:28.381684 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:28.381661 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl"] Jul 14 18:52:28.387237 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:28.387215 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-d9jvl"] Jul 14 18:52:29.782663 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:52:29.782627 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" path="/var/lib/kubelet/pods/da6ac338-c6fc-47f9-a15b-c12f377d1a8f/volumes" Jul 14 18:53:06.333828 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.333793 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v"] Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334101 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334112 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334121 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334128 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334137 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334143 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334196 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334204 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="e23660e6-d2fa-40ca-be99-b93bba8981ad" containerName="istio-proxy" Jul 14 18:53:06.334229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.334210 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:53:06.337229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.337213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:06.340035 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.340007 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:53:06.340425 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.340406 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:53:06.352326 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.352305 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v"] Jul 14 18:53:06.438420 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.438389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6347ff-6d5e-43be-bdfc-30845afdcd8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v\" (UID: \"8f6347ff-6d5e-43be-bdfc-30845afdcd8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:06.539248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.539220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6347ff-6d5e-43be-bdfc-30845afdcd8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v\" (UID: \"8f6347ff-6d5e-43be-bdfc-30845afdcd8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:06.539858 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.539839 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6347ff-6d5e-43be-bdfc-30845afdcd8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v\" (UID: \"8f6347ff-6d5e-43be-bdfc-30845afdcd8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:06.647625 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.647535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:06.773743 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:06.773711 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v"] Jul 14 18:53:06.776218 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:53:06.776192 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f6347ff_6d5e_43be_bdfc_30845afdcd8a.slice/crio-58b516502fcdd5638b236d6f24e5669617d8cc6619f5c2dbf1d9c5d8433cb5f8 WatchSource:0}: Error finding container 58b516502fcdd5638b236d6f24e5669617d8cc6619f5c2dbf1d9c5d8433cb5f8: Status 404 returned error can't find the container with id 58b516502fcdd5638b236d6f24e5669617d8cc6619f5c2dbf1d9c5d8433cb5f8 Jul 14 18:53:07.533107 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:07.533076 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f6347ff-6d5e-43be-bdfc-30845afdcd8a" containerID="245cfccc5e5ed5b91c115d7133a9c3471a6786a38660d8dc473b8b34046bfd89" exitCode=0 Jul 14 18:53:07.533421 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:07.533145 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" event={"ID":"8f6347ff-6d5e-43be-bdfc-30845afdcd8a","Type":"ContainerDied","Data":"245cfccc5e5ed5b91c115d7133a9c3471a6786a38660d8dc473b8b34046bfd89"} Jul 14 18:53:07.533421 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:07.533182 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" event={"ID":"8f6347ff-6d5e-43be-bdfc-30845afdcd8a","Type":"ContainerStarted","Data":"58b516502fcdd5638b236d6f24e5669617d8cc6619f5c2dbf1d9c5d8433cb5f8"} Jul 14 18:53:08.663899 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:08.663874 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:08.759514 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:08.759482 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6347ff-6d5e-43be-bdfc-30845afdcd8a-maas-api-service-ca\") pod \"8f6347ff-6d5e-43be-bdfc-30845afdcd8a\" (UID: \"8f6347ff-6d5e-43be-bdfc-30845afdcd8a\") " Jul 14 18:53:08.759844 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:08.759820 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f6347ff-6d5e-43be-bdfc-30845afdcd8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8f6347ff-6d5e-43be-bdfc-30845afdcd8a" (UID: "8f6347ff-6d5e-43be-bdfc-30845afdcd8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:53:08.860436 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:08.860360 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f6347ff-6d5e-43be-bdfc-30845afdcd8a-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:09.543819 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:09.543786 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" event={"ID":"8f6347ff-6d5e-43be-bdfc-30845afdcd8a","Type":"ContainerDied","Data":"58b516502fcdd5638b236d6f24e5669617d8cc6619f5c2dbf1d9c5d8433cb5f8"} Jul 14 18:53:09.543993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:09.543843 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58b516502fcdd5638b236d6f24e5669617d8cc6619f5c2dbf1d9c5d8433cb5f8" Jul 14 18:53:09.543993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:09.543803 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v" Jul 14 18:53:31.622152 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:31.622118 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5"] Jul 14 18:53:31.622651 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:31.622347 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" containerID="cri-o://114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e" gracePeriod=30 Jul 14 18:53:33.926558 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:33.926518 2580 patch_prober.go:28] interesting pod/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 18:53:33.926959 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:33.926611 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:53:36.869269 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:36.869242 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:53:37.001184 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001151 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-envoy\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001184 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001187 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-data\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001209 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-credential-socket\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001239 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-socket\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001277 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-podinfo\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001345 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-token\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001378 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ddb87\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-kube-api-access-ddb87\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001402 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istiod-ca-cert\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001448 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-certs\") pod \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\" (UID: \"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0\") " Jul 14 18:53:37.001685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001510 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:37.001685 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001621 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:37.001927 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001734 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:37.001927 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001825 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.001927 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001857 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.002090 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.001941 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:53:37.002171 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.002152 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:37.003485 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.003460 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:53:37.003709 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.003691 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:53:37.004129 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.004103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:53:37.004129 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.004116 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-kube-api-access-ddb87" (OuterVolumeSpecName: "kube-api-access-ddb87") pod "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" (UID: "8f72fe17-86b3-4f67-bae4-c3cad56ba1b0"). InnerVolumeSpecName "kube-api-access-ddb87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:53:37.102550 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102520 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.102550 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102547 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddb87\" (UniqueName: \"kubernetes.io/projected/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-kube-api-access-ddb87\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.102550 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102557 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.102794 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102566 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.102794 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102574 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.102794 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102581 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.102794 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.102588 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:53:37.673744 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.673704 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerID="114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e" exitCode=0 Jul 14 18:53:37.673947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.673776 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" Jul 14 18:53:37.673947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.673795 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" event={"ID":"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0","Type":"ContainerDied","Data":"114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e"} Jul 14 18:53:37.673947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.673865 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5" event={"ID":"8f72fe17-86b3-4f67-bae4-c3cad56ba1b0","Type":"ContainerDied","Data":"b7ca1d212094e186e0032bd356bd04ea6b5d341ce056bdd4f59b5a73feaf7a93"} Jul 14 18:53:37.673947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.673892 2580 scope.go:117] "RemoveContainer" containerID="114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e" Jul 14 18:53:37.684712 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.684693 2580 scope.go:117] "RemoveContainer" containerID="83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812" Jul 14 18:53:37.692956 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.692937 2580 scope.go:117] "RemoveContainer" containerID="114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e" Jul 14 18:53:37.693227 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:53:37.693209 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e\": container with ID starting with 114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e not found: ID does not exist" containerID="114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e" Jul 14 18:53:37.693281 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.693237 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e"} err="failed to get container status \"114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e\": rpc error: code = NotFound desc = could not find container \"114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e\": container with ID starting with 114511a6852c00c642f3b734ede541a5cc199fa391045d1342195361ac9d051e not found: ID does not exist" Jul 14 18:53:37.693281 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.693255 2580 scope.go:117] "RemoveContainer" containerID="83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812" Jul 14 18:53:37.693490 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:53:37.693472 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812\": container with ID starting with 83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812 not found: ID does not exist" containerID="83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812" Jul 14 18:53:37.693538 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.693497 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812"} err="failed to get container status \"83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812\": rpc error: code = NotFound desc = could not find container \"83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812\": container with ID starting with 83ef58663be4f9e001d38ca24d1daf58ea38330f6afc1f3309dffa056ddab812 not found: ID does not exist" Jul 14 18:53:37.701046 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.701021 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5"] Jul 14 18:53:37.707406 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.707386 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a8b422c9-openshift-default-8455bc9655-wffn5"] Jul 14 18:53:37.783311 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:53:37.783267 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" path="/var/lib/kubelet/pods/8f72fe17-86b3-4f67-bae4-c3cad56ba1b0/volumes" Jul 14 18:55:12.804701 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.804617 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc"] Jul 14 18:55:12.805286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805216 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:55:12.805286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805236 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:55:12.805286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805257 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" Jul 14 18:55:12.805286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805266 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" Jul 14 18:55:12.805286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805274 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f6347ff-6d5e-43be-bdfc-30845afdcd8a" containerName="revoke-keys" Jul 14 18:55:12.805286 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805283 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f6347ff-6d5e-43be-bdfc-30845afdcd8a" containerName="revoke-keys" Jul 14 18:55:12.805593 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805358 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="da6ac338-c6fc-47f9-a15b-c12f377d1a8f" containerName="revoke-keys" Jul 14 18:55:12.805593 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805372 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" Jul 14 18:55:12.805593 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805382 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" Jul 14 18:55:12.805593 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.805391 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f6347ff-6d5e-43be-bdfc-30845afdcd8a" containerName="revoke-keys" Jul 14 18:55:12.808191 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.808172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.810732 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.810713 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-34c47f4d-openshift-default-dockercfg-kwkgx\"" Jul 14 18:55:12.838980 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.838957 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc"] Jul 14 18:55:12.900952 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.900916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-envoy\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901137 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.900964 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456fc\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-kube-api-access-456fc\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901137 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-socket\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901137 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901068 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-data\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901137 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901100 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ffde57-bfad-4a9f-87dc-cd6981269314-istiod-ca-cert\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901185 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-podinfo\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901246 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-token\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901290 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-certs\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:12.901362 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:12.901319 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-credential-socket\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001756 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001725 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ffde57-bfad-4a9f-87dc-cd6981269314-istiod-ca-cert\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001786 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-podinfo\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001835 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-token\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-certs\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001889 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-credential-socket\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-envoy\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001930 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456fc\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-kube-api-access-456fc\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.001964 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.001961 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-socket\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.002389 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.002006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-data\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.002469 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.002440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-data\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.002529 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.002487 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-certs\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.002586 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.002531 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-socket\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.002586 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.002567 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ffde57-bfad-4a9f-87dc-cd6981269314-istiod-ca-cert\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.002810 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.002754 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-credential-socket\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.004282 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.004255 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-envoy\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.004792 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.004747 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-podinfo\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.012658 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.012612 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-token\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.012827 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.012684 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-456fc\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-kube-api-access-456fc\") pod \"e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.118121 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.118044 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:13.259032 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.259005 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc"] Jul 14 18:55:13.260655 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:55:13.260627 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96ffde57_bfad_4a9f_87dc_cd6981269314.slice/crio-8036a6f2abb6b88c622f980475c7cf464e1384895e93de34098557b46d782f71 WatchSource:0}: Error finding container 8036a6f2abb6b88c622f980475c7cf464e1384895e93de34098557b46d782f71: Status 404 returned error can't find the container with id 8036a6f2abb6b88c622f980475c7cf464e1384895e93de34098557b46d782f71 Jul 14 18:55:13.262712 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.262681 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:55:13.262803 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.262745 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:55:13.262803 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:13.262794 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:55:14.087669 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:14.087637 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" event={"ID":"96ffde57-bfad-4a9f-87dc-cd6981269314","Type":"ContainerStarted","Data":"2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510"} Jul 14 18:55:14.088045 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:14.087697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" event={"ID":"96ffde57-bfad-4a9f-87dc-cd6981269314","Type":"ContainerStarted","Data":"8036a6f2abb6b88c622f980475c7cf464e1384895e93de34098557b46d782f71"} Jul 14 18:55:14.110315 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:14.110269 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podStartSLOduration=2.110254791 podStartE2EDuration="2.110254791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:55:14.108642126 +0000 UTC m=+2004.833782188" watchObservedRunningTime="2026-07-14 18:55:14.110254791 +0000 UTC m=+2004.835394851" Jul 14 18:55:14.118625 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:14.118602 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:14.123991 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:14.123970 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:15.092726 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:15.092691 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:15.093808 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:15.093788 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:28.752713 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.752675 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz"] Jul 14 18:55:28.753128 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.753062 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" Jul 14 18:55:28.753128 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.753075 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f72fe17-86b3-4f67-bae4-c3cad56ba1b0" containerName="istio-proxy" Jul 14 18:55:28.756956 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.756939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:28.759806 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.759782 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:55:28.759951 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.759909 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:55:28.767788 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.767745 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz"] Jul 14 18:55:28.834063 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.834030 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62feab2e-4c2e-41fa-ab11-8822302852b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz\" (UID: \"62feab2e-4c2e-41fa-ab11-8822302852b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:28.935469 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.935435 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62feab2e-4c2e-41fa-ab11-8822302852b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz\" (UID: \"62feab2e-4c2e-41fa-ab11-8822302852b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:28.936105 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:28.936081 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62feab2e-4c2e-41fa-ab11-8822302852b7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz\" (UID: \"62feab2e-4c2e-41fa-ab11-8822302852b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:29.068481 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:29.068446 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:29.209429 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:29.209239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz"] Jul 14 18:55:29.212024 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:55:29.211982 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62feab2e_4c2e_41fa_ab11_8822302852b7.slice/crio-ec6189e083b78e6874c473b30b2e92fbbe147c8835582fbb6756cdce2f0aa71e WatchSource:0}: Error finding container ec6189e083b78e6874c473b30b2e92fbbe147c8835582fbb6756cdce2f0aa71e: Status 404 returned error can't find the container with id ec6189e083b78e6874c473b30b2e92fbbe147c8835582fbb6756cdce2f0aa71e Jul 14 18:55:30.094484 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.094442 2580 patch_prober.go:28] interesting pod/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 14 18:55:30.094895 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.094501 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 14 18:55:30.174165 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.174132 2580 generic.go:356] "Generic (PLEG): container finished" podID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerID="2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510" exitCode=0 Jul 14 18:55:30.174355 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.174214 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" event={"ID":"96ffde57-bfad-4a9f-87dc-cd6981269314","Type":"ContainerDied","Data":"2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510"} Jul 14 18:55:30.174628 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.174611 2580 scope.go:117] "RemoveContainer" containerID="2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510" Jul 14 18:55:30.175776 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.175725 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:55:30.175870 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.175774 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" event={"ID":"62feab2e-4c2e-41fa-ab11-8822302852b7","Type":"ContainerDied","Data":"be1990596cbe4a5c74f67ab6aafea8aa42ed145db5a6936bceff8bad465cd6aa"} Jul 14 18:55:30.175870 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.175844 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:55:30.175948 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.175739 2580 generic.go:356] "Generic (PLEG): container finished" podID="62feab2e-4c2e-41fa-ab11-8822302852b7" containerID="be1990596cbe4a5c74f67ab6aafea8aa42ed145db5a6936bceff8bad465cd6aa" exitCode=0 Jul 14 18:55:30.175948 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.175882 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:55:30.175948 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:30.175937 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" event={"ID":"62feab2e-4c2e-41fa-ab11-8822302852b7","Type":"ContainerStarted","Data":"ec6189e083b78e6874c473b30b2e92fbbe147c8835582fbb6756cdce2f0aa71e"} Jul 14 18:55:31.182363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:31.182326 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" event={"ID":"96ffde57-bfad-4a9f-87dc-cd6981269314","Type":"ContainerStarted","Data":"bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3"} Jul 14 18:55:31.356689 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:31.356657 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:31.457458 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:31.457371 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62feab2e-4c2e-41fa-ab11-8822302852b7-maas-api-service-ca\") pod \"62feab2e-4c2e-41fa-ab11-8822302852b7\" (UID: \"62feab2e-4c2e-41fa-ab11-8822302852b7\") " Jul 14 18:55:31.457743 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:31.457715 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62feab2e-4c2e-41fa-ab11-8822302852b7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62feab2e-4c2e-41fa-ab11-8822302852b7" (UID: "62feab2e-4c2e-41fa-ab11-8822302852b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:31.558190 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:31.558149 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62feab2e-4c2e-41fa-ab11-8822302852b7-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:55:32.118638 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:32.118539 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:32.118884 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:32.118860 2580 patch_prober.go:28] interesting pod/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 14 18:55:32.118935 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:32.118908 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 14 18:55:32.188055 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:32.188023 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" Jul 14 18:55:32.188055 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:32.188024 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz" event={"ID":"62feab2e-4c2e-41fa-ab11-8822302852b7","Type":"ContainerDied","Data":"ec6189e083b78e6874c473b30b2e92fbbe147c8835582fbb6756cdce2f0aa71e"} Jul 14 18:55:32.188489 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:32.188091 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6189e083b78e6874c473b30b2e92fbbe147c8835582fbb6756cdce2f0aa71e" Jul 14 18:55:33.119535 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:33.119495 2580 patch_prober.go:28] interesting pod/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 14 18:55:33.119721 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:33.119555 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 14 18:55:34.119264 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:34.119231 2580 patch_prober.go:28] interesting pod/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 14 18:55:34.119651 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:34.119284 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 14 18:55:35.118880 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:35.118843 2580 patch_prober.go:28] interesting pod/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 14 18:55:35.119055 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:35.118902 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 14 18:55:36.122519 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:36.122492 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:36.122918 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:36.122775 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:36.123502 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:36.123479 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:49.820248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:49.820216 2580 scope.go:117] "RemoveContainer" containerID="536dc610888af0c965c4d544882262ad1914441781a6efc02985e32aa41a5b33" Jul 14 18:55:54.588828 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:54.588792 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc"] Jul 14 18:55:54.589299 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:54.588998 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" containerID="cri-o://bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3" gracePeriod=30 Jul 14 18:55:59.840297 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.840270 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:55:59.916192 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916111 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-data\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916192 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916151 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-podinfo\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916192 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916171 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-envoy\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916206 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-credential-socket\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916258 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-token\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916283 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-socket\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916322 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ffde57-bfad-4a9f-87dc-cd6981269314-istiod-ca-cert\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916344 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-certs\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916440 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916371 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-456fc\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-kube-api-access-456fc\") pod \"96ffde57-bfad-4a9f-87dc-cd6981269314\" (UID: \"96ffde57-bfad-4a9f-87dc-cd6981269314\") " Jul 14 18:55:59.916751 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916591 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:59.916751 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916728 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96ffde57-bfad-4a9f-87dc-cd6981269314-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:55:59.916897 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916818 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:59.916897 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.916856 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:59.917108 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.917084 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-data" (OuterVolumeSpecName: "istio-data") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:59.918689 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.918661 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:55:59.918814 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.918683 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-token" (OuterVolumeSpecName: "istio-token") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:55:59.918814 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.918732 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:55:59.918977 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:55:59.918957 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-kube-api-access-456fc" (OuterVolumeSpecName: "kube-api-access-456fc") pod "96ffde57-bfad-4a9f-87dc-cd6981269314" (UID: "96ffde57-bfad-4a9f-87dc-cd6981269314"). InnerVolumeSpecName "kube-api-access-456fc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:56:00.017086 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017051 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-456fc\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-kube-api-access-456fc\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017086 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017080 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017086 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017089 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017086 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017096 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017106 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017116 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96ffde57-bfad-4a9f-87dc-cd6981269314-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017126 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017135 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96ffde57-bfad-4a9f-87dc-cd6981269314-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.017325 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.017142 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96ffde57-bfad-4a9f-87dc-cd6981269314-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:00.311263 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.311220 2580 generic.go:356] "Generic (PLEG): container finished" podID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerID="bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3" exitCode=0 Jul 14 18:56:00.311263 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.311252 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" event={"ID":"96ffde57-bfad-4a9f-87dc-cd6981269314","Type":"ContainerDied","Data":"bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3"} Jul 14 18:56:00.311470 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.311279 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" Jul 14 18:56:00.311470 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.311347 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc" event={"ID":"96ffde57-bfad-4a9f-87dc-cd6981269314","Type":"ContainerDied","Data":"8036a6f2abb6b88c622f980475c7cf464e1384895e93de34098557b46d782f71"} Jul 14 18:56:00.311470 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.311373 2580 scope.go:117] "RemoveContainer" containerID="bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3" Jul 14 18:56:00.322363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.322341 2580 scope.go:117] "RemoveContainer" containerID="2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510" Jul 14 18:56:00.330030 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.330011 2580 scope.go:117] "RemoveContainer" containerID="bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3" Jul 14 18:56:00.330294 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:56:00.330275 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3\": container with ID starting with bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3 not found: ID does not exist" containerID="bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3" Jul 14 18:56:00.330339 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.330305 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3"} err="failed to get container status \"bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3\": rpc error: code = NotFound desc = could not find container \"bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3\": container with ID starting with bf5b3cd7652f3ed79ff2d868ca3be2b67ea491dd65dbf43b5ce81c49dbe7aef3 not found: ID does not exist" Jul 14 18:56:00.330339 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.330322 2580 scope.go:117] "RemoveContainer" containerID="2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510" Jul 14 18:56:00.330550 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:56:00.330533 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510\": container with ID starting with 2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510 not found: ID does not exist" containerID="2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510" Jul 14 18:56:00.330591 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.330557 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510"} err="failed to get container status \"2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510\": rpc error: code = NotFound desc = could not find container \"2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510\": container with ID starting with 2179e4aee669b83f56628ea1d6d7c8ddb317b20cb7a3e8e25987eb3ea7e41510 not found: ID does not exist" Jul 14 18:56:00.336025 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.336004 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc"] Jul 14 18:56:00.343598 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:00.343578 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34c47f4d-openshift-default-546dc96855-9cznc"] Jul 14 18:56:01.783644 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:01.783609 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" path="/var/lib/kubelet/pods/96ffde57-bfad-4a9f-87dc-cd6981269314/volumes" Jul 14 18:56:16.008589 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.008498 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h"] Jul 14 18:56:16.010021 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.009990 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62feab2e-4c2e-41fa-ab11-8822302852b7" containerName="revoke-keys" Jul 14 18:56:16.010021 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010021 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="62feab2e-4c2e-41fa-ab11-8822302852b7" containerName="revoke-keys" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010033 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010042 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010061 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010070 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010143 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="62feab2e-4c2e-41fa-ab11-8822302852b7" containerName="revoke-keys" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010157 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" Jul 14 18:56:16.010195 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.010165 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="96ffde57-bfad-4a9f-87dc-cd6981269314" containerName="istio-proxy" Jul 14 18:56:16.015051 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.015030 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.018894 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.018870 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-99a89b27-openshift-default-dockercfg-h7xxj\"" Jul 14 18:56:16.035209 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.035185 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h"] Jul 14 18:56:16.042861 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.042835 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrhz9\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-kube-api-access-vrhz9\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.042957 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.042886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-data\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043003 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.042966 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-podinfo\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043039 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.043012 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-credential-socket\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043074 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.043053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-certs\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043110 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.043084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-token\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043152 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.043113 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724e174-c5de-4516-80ee-102e2c4c3a34-istiod-ca-cert\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043228 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.043209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-socket\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.043266 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.043256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-envoy\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144409 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-token\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144600 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724e174-c5de-4516-80ee-102e2c4c3a34-istiod-ca-cert\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144600 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144484 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-socket\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144600 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144520 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-envoy\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144600 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrhz9\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-kube-api-access-vrhz9\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144854 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144695 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-data\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144854 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-podinfo\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144854 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144836 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-credential-socket\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.144998 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.144865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-certs\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.145295 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.145062 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-data\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.145295 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.145241 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-socket\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.145295 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.145286 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-certs\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.145456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.145334 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724e174-c5de-4516-80ee-102e2c4c3a34-istiod-ca-cert\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.145574 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.145552 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-credential-socket\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.147036 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.147015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-envoy\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.147395 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.147377 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-podinfo\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.156177 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.156153 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-token\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.163540 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.163518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrhz9\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-kube-api-access-vrhz9\") pod \"e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.325146 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.325115 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:16.472167 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.472135 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h"] Jul 14 18:56:16.475179 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:56:16.475153 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4724e174_c5de_4516_80ee_102e2c4c3a34.slice/crio-a11c2ab7cf58487497bf58d1c0f4ec995911c7e17bad870380e133f90ec78eed WatchSource:0}: Error finding container a11c2ab7cf58487497bf58d1c0f4ec995911c7e17bad870380e133f90ec78eed: Status 404 returned error can't find the container with id a11c2ab7cf58487497bf58d1c0f4ec995911c7e17bad870380e133f90ec78eed Jul 14 18:56:16.477231 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.477198 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:56:16.477307 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.477269 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:56:16.477307 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:16.477297 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:56:17.387832 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:17.387792 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" event={"ID":"4724e174-c5de-4516-80ee-102e2c4c3a34","Type":"ContainerStarted","Data":"1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d"} Jul 14 18:56:17.388320 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:17.387845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" event={"ID":"4724e174-c5de-4516-80ee-102e2c4c3a34","Type":"ContainerStarted","Data":"a11c2ab7cf58487497bf58d1c0f4ec995911c7e17bad870380e133f90ec78eed"} Jul 14 18:56:17.417788 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:17.417715 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" podStartSLOduration=2.417699325 podStartE2EDuration="2.417699325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:17.415910773 +0000 UTC m=+2068.141050846" watchObservedRunningTime="2026-07-14 18:56:17.417699325 +0000 UTC m=+2068.142839385" Jul 14 18:56:18.325604 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:18.325568 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:18.330456 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:18.330431 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:18.392293 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:18.392264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:18.393284 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:18.393264 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:33.616653 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:33.616244 2580 patch_prober.go:28] interesting pod/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:55606->10.134.0.38:15021: read: connection reset by peer" start-of-body= Jul 14 18:56:33.616653 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:33.616332 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": read tcp 10.134.0.2:55606->10.134.0.38:15021: read: connection reset by peer" Jul 14 18:56:34.475418 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:34.475380 2580 generic.go:356] "Generic (PLEG): container finished" podID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerID="1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d" exitCode=0 Jul 14 18:56:34.475599 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:34.475415 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" event={"ID":"4724e174-c5de-4516-80ee-102e2c4c3a34","Type":"ContainerDied","Data":"1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d"} Jul 14 18:56:34.475920 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:34.475902 2580 scope.go:117] "RemoveContainer" containerID="1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d" Jul 14 18:56:34.477052 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:34.477022 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:56:34.477141 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:34.477126 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:56:34.477188 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:34.477166 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:56:35.482528 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:35.482490 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" event={"ID":"4724e174-c5de-4516-80ee-102e2c4c3a34","Type":"ContainerStarted","Data":"c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7"} Jul 14 18:56:36.325958 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:36.325922 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:36.326222 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:36.326190 2580 patch_prober.go:28] interesting pod/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 14 18:56:36.326444 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:36.326418 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 14 18:56:37.325975 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:37.325933 2580 patch_prober.go:28] interesting pod/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 14 18:56:37.326469 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:37.326014 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 14 18:56:38.326248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:38.325805 2580 patch_prober.go:28] interesting pod/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body= Jul 14 18:56:38.326248 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:38.325876 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" Jul 14 18:56:39.329629 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:39.329595 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:39.330148 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:39.330125 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:39.330723 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:39.330704 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:56:49.753530 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:49.753503 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:56:49.755331 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:49.755310 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 18:56:52.413357 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.413321 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4"] Jul 14 18:56:52.417699 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.417683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:52.420240 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.420218 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 14 18:56:52.420381 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.420364 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lvrkw\"" Jul 14 18:56:52.423699 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.423674 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4"] Jul 14 18:56:52.563350 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.563314 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df4563-c20d-4a37-a406-6cd8fa4dd9f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4\" (UID: \"87df4563-c20d-4a37-a406-6cd8fa4dd9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:52.664067 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.663962 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df4563-c20d-4a37-a406-6cd8fa4dd9f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4\" (UID: \"87df4563-c20d-4a37-a406-6cd8fa4dd9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:52.664586 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.664566 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df4563-c20d-4a37-a406-6cd8fa4dd9f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4\" (UID: \"87df4563-c20d-4a37-a406-6cd8fa4dd9f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:52.730013 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.729987 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:52.859368 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:52.859341 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4"] Jul 14 18:56:52.861997 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:56:52.861962 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87df4563_c20d_4a37_a406_6cd8fa4dd9f4.slice/crio-68f5efaa8b9be168cbc0bb4e6b2d48b5a929865f6114d3ad28f9f0d2cb19c400 WatchSource:0}: Error finding container 68f5efaa8b9be168cbc0bb4e6b2d48b5a929865f6114d3ad28f9f0d2cb19c400: Status 404 returned error can't find the container with id 68f5efaa8b9be168cbc0bb4e6b2d48b5a929865f6114d3ad28f9f0d2cb19c400 Jul 14 18:56:53.568361 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:53.568324 2580 generic.go:356] "Generic (PLEG): container finished" podID="87df4563-c20d-4a37-a406-6cd8fa4dd9f4" containerID="960d10df3d542268de83e57851003406dda2a601928c5c48dacd8d2ac2fa57d3" exitCode=0 Jul 14 18:56:53.568707 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:53.568375 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" event={"ID":"87df4563-c20d-4a37-a406-6cd8fa4dd9f4","Type":"ContainerDied","Data":"960d10df3d542268de83e57851003406dda2a601928c5c48dacd8d2ac2fa57d3"} Jul 14 18:56:53.568707 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:53.568444 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" event={"ID":"87df4563-c20d-4a37-a406-6cd8fa4dd9f4","Type":"ContainerStarted","Data":"68f5efaa8b9be168cbc0bb4e6b2d48b5a929865f6114d3ad28f9f0d2cb19c400"} Jul 14 18:56:54.698909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:54.698883 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:54.882634 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:54.882546 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df4563-c20d-4a37-a406-6cd8fa4dd9f4-maas-api-service-ca\") pod \"87df4563-c20d-4a37-a406-6cd8fa4dd9f4\" (UID: \"87df4563-c20d-4a37-a406-6cd8fa4dd9f4\") " Jul 14 18:56:54.882930 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:54.882910 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87df4563-c20d-4a37-a406-6cd8fa4dd9f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87df4563-c20d-4a37-a406-6cd8fa4dd9f4" (UID: "87df4563-c20d-4a37-a406-6cd8fa4dd9f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:56:54.983545 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:54.983517 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87df4563-c20d-4a37-a406-6cd8fa4dd9f4-maas-api-service-ca\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:56:55.578636 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:55.578605 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" Jul 14 18:56:55.578819 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:55.578605 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4" event={"ID":"87df4563-c20d-4a37-a406-6cd8fa4dd9f4","Type":"ContainerDied","Data":"68f5efaa8b9be168cbc0bb4e6b2d48b5a929865f6114d3ad28f9f0d2cb19c400"} Jul 14 18:56:55.578819 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:56:55.578732 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68f5efaa8b9be168cbc0bb4e6b2d48b5a929865f6114d3ad28f9f0d2cb19c400" Jul 14 18:57:49.848851 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:57:49.848746 2580 scope.go:117] "RemoveContainer" containerID="1006cd3587253377155c57de108853f754233a6ed13f0b0a2c7870c45f606984" Jul 14 18:58:00.058776 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:00.058729 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h"] Jul 14 18:58:00.059283 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:00.058970 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" containerID="cri-o://c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7" gracePeriod=30 Jul 14 18:58:05.318229 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.318197 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:58:05.357041 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.356969 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-credential-socket\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357041 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357007 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-token\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357239 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357041 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-socket\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357239 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357081 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-certs\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357239 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357141 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-data\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357239 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357167 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-envoy\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357239 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357205 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-podinfo\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357490 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357253 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrhz9\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-kube-api-access-vrhz9\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357490 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357282 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:58:05.357490 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724e174-c5de-4516-80ee-102e2c4c3a34-istiod-ca-cert\") pod \"4724e174-c5de-4516-80ee-102e2c4c3a34\" (UID: \"4724e174-c5de-4516-80ee-102e2c4c3a34\") " Jul 14 18:58:05.357490 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357411 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:58:05.357490 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357458 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:58:05.357749 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357616 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.357749 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357639 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.357749 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357654 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.357749 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.357731 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4724e174-c5de-4516-80ee-102e2c4c3a34-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:58:05.358621 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.358585 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-data" (OuterVolumeSpecName: "istio-data") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:58:05.359659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.359634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:58:05.359659 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.359652 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-token" (OuterVolumeSpecName: "istio-token") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:58:05.359813 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.359634 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 18:58:05.359953 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.359934 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-kube-api-access-vrhz9" (OuterVolumeSpecName: "kube-api-access-vrhz9") pod "4724e174-c5de-4516-80ee-102e2c4c3a34" (UID: "4724e174-c5de-4516-80ee-102e2c4c3a34"). InnerVolumeSpecName "kube-api-access-vrhz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:58:05.458097 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.458057 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4724e174-c5de-4516-80ee-102e2c4c3a34-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.458097 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.458087 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.458097 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.458095 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.458097 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.458105 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.458356 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.458113 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4724e174-c5de-4516-80ee-102e2c4c3a34-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.458356 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.458121 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrhz9\" (UniqueName: \"kubernetes.io/projected/4724e174-c5de-4516-80ee-102e2c4c3a34-kube-api-access-vrhz9\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 18:58:05.894215 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.894181 2580 generic.go:356] "Generic (PLEG): container finished" podID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerID="c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7" exitCode=0 Jul 14 18:58:05.894394 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.894247 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" event={"ID":"4724e174-c5de-4516-80ee-102e2c4c3a34","Type":"ContainerDied","Data":"c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7"} Jul 14 18:58:05.894394 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.894295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" event={"ID":"4724e174-c5de-4516-80ee-102e2c4c3a34","Type":"ContainerDied","Data":"a11c2ab7cf58487497bf58d1c0f4ec995911c7e17bad870380e133f90ec78eed"} Jul 14 18:58:05.894394 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.894321 2580 scope.go:117] "RemoveContainer" containerID="c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7" Jul 14 18:58:05.894394 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.894240 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h" Jul 14 18:58:05.902878 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.902849 2580 scope.go:117] "RemoveContainer" containerID="1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d" Jul 14 18:58:05.910494 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.910476 2580 scope.go:117] "RemoveContainer" containerID="c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7" Jul 14 18:58:05.910727 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:58:05.910704 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7\": container with ID starting with c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7 not found: ID does not exist" containerID="c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7" Jul 14 18:58:05.910787 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.910737 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7"} err="failed to get container status \"c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7\": rpc error: code = NotFound desc = could not find container \"c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7\": container with ID starting with c4ac29c7302132af089bd38b19d2b744f42a2fd2c7bb0bb1b18cc0f6acc12cf7 not found: ID does not exist" Jul 14 18:58:05.910839 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.910754 2580 scope.go:117] "RemoveContainer" containerID="1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d" Jul 14 18:58:05.911061 ip-10-0-131-199 kubenswrapper[2580]: E0714 18:58:05.911044 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d\": container with ID starting with 1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d not found: ID does not exist" containerID="1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d" Jul 14 18:58:05.911116 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.911068 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d"} err="failed to get container status \"1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d\": rpc error: code = NotFound desc = could not find container \"1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d\": container with ID starting with 1d046672891f8e81246b43a21d173d103020524824e43e935d6e1815e284551d not found: ID does not exist" Jul 14 18:58:05.914337 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.914312 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h"] Jul 14 18:58:05.918273 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:05.918253 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-99a89b27-openshift-default-5888b988c-tbh2h"] Jul 14 18:58:07.783080 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:07.783050 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" path="/var/lib/kubelet/pods/4724e174-c5de-4516-80ee-102e2c4c3a34/volumes" Jul 14 18:58:49.867921 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:58:49.867885 2580 scope.go:117] "RemoveContainer" containerID="d36a602c24ff1648e88791434e9b7cd189cc0bf5e4c4f2b33738264e72d20816" Jul 14 18:59:01.992174 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992136 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp"] Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992579 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992598 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992636 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87df4563-c20d-4a37-a406-6cd8fa4dd9f4" containerName="revoke-keys" Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992645 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="87df4563-c20d-4a37-a406-6cd8fa4dd9f4" containerName="revoke-keys" Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992714 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="87df4563-c20d-4a37-a406-6cd8fa4dd9f4" containerName="revoke-keys" Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992728 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" Jul 14 18:59:01.992754 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.992740 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" Jul 14 18:59:01.995789 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.995744 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:01.998438 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:01.998414 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0895b5-openshift-default-dockercfg-s6b64\"" Jul 14 18:59:02.011870 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.011842 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp"] Jul 14 18:59:02.104427 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104396 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istiod-ca-cert\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104427 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104430 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-podinfo\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104675 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104457 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-token\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104675 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104569 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-data\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104675 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-envoy\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104675 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104671 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-credential-socket\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104911 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104709 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfd4x\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-kube-api-access-vfd4x\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104911 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104750 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-socket\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.104911 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.104822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-certs\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205249 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205211 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-certs\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205438 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istiod-ca-cert\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205438 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205382 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-podinfo\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205438 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-token\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205469 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-data\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205504 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-envoy\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205556 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-credential-socket\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205588 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfd4x\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-kube-api-access-vfd4x\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205619 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205599 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-certs\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205907 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205633 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-socket\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205907 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205822 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-data\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.205907 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.205882 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-credential-socket\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.206066 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.206015 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-socket\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.206066 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.206058 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istiod-ca-cert\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.207993 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.207975 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-envoy\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.208080 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.208010 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-podinfo\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.214477 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.214449 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-token\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.214631 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.214613 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfd4x\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-kube-api-access-vfd4x\") pod \"e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.306209 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.306172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:02.447300 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.447272 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp"] Jul 14 18:59:02.449618 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:59:02.449590 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa78d8aa_ed1f_4360_ab52_cfce611e5f3f.slice/crio-3e63b27011a5abd5f14295c26157e7c33234fe993614b6ce142f120fdd6c7dfb WatchSource:0}: Error finding container 3e63b27011a5abd5f14295c26157e7c33234fe993614b6ce142f120fdd6c7dfb: Status 404 returned error can't find the container with id 3e63b27011a5abd5f14295c26157e7c33234fe993614b6ce142f120fdd6c7dfb Jul 14 18:59:02.451921 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.451881 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:02.451995 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.451966 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:02.452044 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:02.452005 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:03.143891 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:03.143851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" event={"ID":"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f","Type":"ContainerStarted","Data":"1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b"} Jul 14 18:59:03.144273 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:03.143945 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" event={"ID":"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f","Type":"ContainerStarted","Data":"3e63b27011a5abd5f14295c26157e7c33234fe993614b6ce142f120fdd6c7dfb"} Jul 14 18:59:03.167416 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:03.167363 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podStartSLOduration=2.167349163 podStartE2EDuration="2.167349163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:03.166052844 +0000 UTC m=+2233.891192909" watchObservedRunningTime="2026-07-14 18:59:03.167349163 +0000 UTC m=+2233.892489226" Jul 14 18:59:03.307365 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:03.307326 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:03.312455 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:03.312429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:04.148363 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:04.148336 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:04.149187 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:04.149170 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:10.367186 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.367150 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch"] Jul 14 18:59:10.367801 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.367512 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" Jul 14 18:59:10.367801 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.367524 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="4724e174-c5de-4516-80ee-102e2c4c3a34" containerName="istio-proxy" Jul 14 18:59:10.370699 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.370681 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.371909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.371884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-socket\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372009 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.371921 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-credential-socket\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372009 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.371990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-token\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372122 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.372027 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-podinfo\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372180 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.372123 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-envoy\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372180 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.372165 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj4c\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-kube-api-access-8xj4c\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372290 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.372199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-certs\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372290 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.372227 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-data\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.372290 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.372256 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istiod-ca-cert\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.375100 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.375078 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-08413e-openshift-default-dockercfg-86gkk\"" Jul 14 18:59:10.383580 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.383562 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch"] Jul 14 18:59:10.472909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.472875 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-token\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.472909 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.472909 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-podinfo\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.472938 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-envoy\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.472959 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj4c\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-kube-api-access-8xj4c\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.472978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-certs\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473000 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-data\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473026 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istiod-ca-cert\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473086 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-socket\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473179 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-credential-socket\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473440 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-certs\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473556 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473518 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-data\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473573 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-socket\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473671 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473620 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-credential-socket\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.473902 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.473881 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istiod-ca-cert\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.475467 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.475443 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-envoy\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.475616 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.475600 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-podinfo\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.483127 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.483104 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-token\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.483578 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.483557 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj4c\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-kube-api-access-8xj4c\") pod \"e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.681121 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.681032 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:10.825403 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.825368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch"] Jul 14 18:59:10.829680 ip-10-0-131-199 kubenswrapper[2580]: W0714 18:59:10.829651 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea67678_5d02_4b0f_a5d5_a6688fd3d19f.slice/crio-324836e1f7f0fb3b762862be00e1290ae9d3fc2765b764a736a227206f4a7bbb WatchSource:0}: Error finding container 324836e1f7f0fb3b762862be00e1290ae9d3fc2765b764a736a227206f4a7bbb: Status 404 returned error can't find the container with id 324836e1f7f0fb3b762862be00e1290ae9d3fc2765b764a736a227206f4a7bbb Jul 14 18:59:10.831838 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.831803 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:10.831926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.831869 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:10.831926 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:10.831899 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:11.180596 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:11.180564 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" event={"ID":"bea67678-5d02-4b0f-a5d5-a6688fd3d19f","Type":"ContainerStarted","Data":"16f175cbad20192a3528ebf72a3845fc8c9f48002e9f849594401830188ee417"} Jul 14 18:59:11.180785 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:11.180617 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" event={"ID":"bea67678-5d02-4b0f-a5d5-a6688fd3d19f","Type":"ContainerStarted","Data":"324836e1f7f0fb3b762862be00e1290ae9d3fc2765b764a736a227206f4a7bbb"} Jul 14 18:59:11.206426 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:11.206373 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podStartSLOduration=1.206358622 podStartE2EDuration="1.206358622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:11.205908715 +0000 UTC m=+2241.931048776" watchObservedRunningTime="2026-07-14 18:59:11.206358622 +0000 UTC m=+2241.931498682" Jul 14 18:59:11.681825 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:11.681751 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:11.687051 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:11.687020 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:12.185972 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:12.185936 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:12.186985 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:12.186961 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:19.427238 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:19.426906 2580 patch_prober.go:28] interesting pod/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:57418->10.134.0.40:15021: read: connection reset by peer" start-of-body= Jul 14 18:59:19.427238 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:19.426970 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": read tcp 10.134.0.2:57418->10.134.0.40:15021: read: connection reset by peer" Jul 14 18:59:20.224186 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:20.224144 2580 generic.go:356] "Generic (PLEG): container finished" podID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerID="1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b" exitCode=0 Jul 14 18:59:20.224354 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:20.224210 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" event={"ID":"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f","Type":"ContainerDied","Data":"1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b"} Jul 14 18:59:20.224712 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:20.224694 2580 scope.go:117] "RemoveContainer" containerID="1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b" Jul 14 18:59:20.225916 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:20.225883 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:20.226016 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:20.225956 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:20.226016 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:20.225988 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:21.255790 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:21.251947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" event={"ID":"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f","Type":"ContainerStarted","Data":"f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc"} Jul 14 18:59:21.306558 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:21.306520 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:21.306802 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:21.306778 2580 patch_prober.go:28] interesting pod/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 14 18:59:21.306868 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:21.306833 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 14 18:59:22.307142 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:22.307099 2580 patch_prober.go:28] interesting pod/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 14 18:59:22.307624 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:22.307168 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 14 18:59:23.307298 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:23.307258 2580 patch_prober.go:28] interesting pod/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 14 18:59:23.307720 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:23.307327 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 14 18:59:24.307189 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:24.307151 2580 patch_prober.go:28] interesting pod/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body= Jul 14 18:59:24.307368 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:24.307221 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" Jul 14 18:59:25.310816 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:25.310783 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:25.311228 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:25.311071 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:25.312034 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:25.312016 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 18:59:27.580606 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:27.580177 2580 patch_prober.go:28] interesting pod/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:51754->10.134.0.41:15021: read: connection reset by peer" start-of-body= Jul 14 18:59:27.580606 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:27.580242 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": read tcp 10.134.0.2:51754->10.134.0.41:15021: read: connection reset by peer" Jul 14 18:59:28.292408 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:28.292374 2580 generic.go:356] "Generic (PLEG): container finished" podID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerID="16f175cbad20192a3528ebf72a3845fc8c9f48002e9f849594401830188ee417" exitCode=0 Jul 14 18:59:28.292584 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:28.292432 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" event={"ID":"bea67678-5d02-4b0f-a5d5-a6688fd3d19f","Type":"ContainerDied","Data":"16f175cbad20192a3528ebf72a3845fc8c9f48002e9f849594401830188ee417"} Jul 14 18:59:28.292854 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:28.292839 2580 scope.go:117] "RemoveContainer" containerID="16f175cbad20192a3528ebf72a3845fc8c9f48002e9f849594401830188ee417" Jul 14 18:59:28.293947 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:28.293915 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:28.294012 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:28.293991 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:28.294056 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:28.294033 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 18:59:29.302314 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:29.302239 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" event={"ID":"bea67678-5d02-4b0f-a5d5-a6688fd3d19f","Type":"ContainerStarted","Data":"59f448c767fe5d564266d76374b3d0974b58cba5b291c0bf6bd32297407d7e86"} Jul 14 18:59:29.682137 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:29.682034 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:29.682324 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:29.682299 2580 patch_prober.go:28] interesting pod/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 14 18:59:29.682409 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:29.682361 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 14 18:59:30.682377 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:30.682338 2580 patch_prober.go:28] interesting pod/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 14 18:59:30.682895 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:30.682402 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 14 18:59:31.682341 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:31.682297 2580 patch_prober.go:28] interesting pod/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 14 18:59:31.682524 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:31.682373 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 14 18:59:32.681882 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:32.681840 2580 patch_prober.go:28] interesting pod/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 14 18:59:32.682097 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:32.681896 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 14 18:59:33.685468 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:33.685439 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:33.685866 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:33.685687 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 18:59:33.686455 ip-10-0-131-199 kubenswrapper[2580]: I0714 18:59:33.686439 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 19:00:33.329678 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:33.329642 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp"] Jul 14 19:00:33.330235 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:33.329930 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" containerID="cri-o://f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc" gracePeriod=30 Jul 14 19:00:38.581203 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.581177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 19:00:38.614243 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.614173 2580 generic.go:356] "Generic (PLEG): container finished" podID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerID="f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc" exitCode=0 Jul 14 19:00:38.614243 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.614227 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" Jul 14 19:00:38.614441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.614251 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" event={"ID":"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f","Type":"ContainerDied","Data":"f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc"} Jul 14 19:00:38.614441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.614305 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp" event={"ID":"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f","Type":"ContainerDied","Data":"3e63b27011a5abd5f14295c26157e7c33234fe993614b6ce142f120fdd6c7dfb"} Jul 14 19:00:38.614441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.614326 2580 scope.go:117] "RemoveContainer" containerID="f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc" Jul 14 19:00:38.622834 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.622818 2580 scope.go:117] "RemoveContainer" containerID="1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b" Jul 14 19:00:38.632519 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.632503 2580 scope.go:117] "RemoveContainer" containerID="f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc" Jul 14 19:00:38.632803 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:00:38.632781 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc\": container with ID starting with f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc not found: ID does not exist" containerID="f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc" Jul 14 19:00:38.632884 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.632815 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc"} err="failed to get container status \"f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc\": rpc error: code = NotFound desc = could not find container \"f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc\": container with ID starting with f570eca56c64f2d6f3269f9ca7e769f663e74aa9f7e3e007386fec5239b97efc not found: ID does not exist" Jul 14 19:00:38.632884 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.632838 2580 scope.go:117] "RemoveContainer" containerID="1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b" Jul 14 19:00:38.633125 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:00:38.633108 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b\": container with ID starting with 1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b not found: ID does not exist" containerID="1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b" Jul 14 19:00:38.633181 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.633130 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b"} err="failed to get container status \"1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b\": rpc error: code = NotFound desc = could not find container \"1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b\": container with ID starting with 1c83bbcb1c2526253042d9d7d8654bc65d1d22764590483a740edd7defbce09b not found: ID does not exist" Jul 14 19:00:38.653866 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.653849 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-socket\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.653946 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.653878 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-credential-socket\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.653946 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.653896 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-data\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.653946 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.653939 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-token\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.654108 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.653965 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-podinfo\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.654108 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.653982 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfd4x\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-kube-api-access-vfd4x\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.654108 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654087 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:38.654263 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654097 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-certs\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.654263 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654172 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:38.654263 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654203 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istiod-ca-cert\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.654263 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654230 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-envoy\") pod \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\" (UID: \"fa78d8aa-ed1f-4360-ab52-cfce611e5f3f\") " Jul 14 19:00:38.654450 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654367 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:38.654544 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654513 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.654544 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654537 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.654685 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654552 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.654685 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.654543 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:00:38.655203 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.655168 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-data" (OuterVolumeSpecName: "istio-data") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:38.656263 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.656227 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:00:38.656374 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.656309 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-kube-api-access-vfd4x" (OuterVolumeSpecName: "kube-api-access-vfd4x") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "kube-api-access-vfd4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:00:38.656374 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.656321 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-token" (OuterVolumeSpecName: "istio-token") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:00:38.656374 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.656335 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" (UID: "fa78d8aa-ed1f-4360-ab52-cfce611e5f3f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:38.754998 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.754966 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.754998 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.754992 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.754998 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.755002 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.755211 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.755009 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.755211 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.755016 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.755211 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.755028 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfd4x\" (UniqueName: \"kubernetes.io/projected/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f-kube-api-access-vfd4x\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:00:38.939706 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.939672 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp"] Jul 14 19:00:38.946639 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:38.946615 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0895b5-openshift-default-85565b58f9-n2rwp"] Jul 14 19:00:39.783021 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:39.782986 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" path="/var/lib/kubelet/pods/fa78d8aa-ed1f-4360-ab52-cfce611e5f3f/volumes" Jul 14 19:00:59.574637 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:59.574605 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch"] Jul 14 19:00:59.575190 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:00:59.574876 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" containerID="cri-o://59f448c767fe5d564266d76374b3d0974b58cba5b291c0bf6bd32297407d7e86" gracePeriod=30 Jul 14 19:01:00.520743 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.520710 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9"] Jul 14 19:01:00.521088 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.521072 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" Jul 14 19:01:00.521135 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.521092 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" Jul 14 19:01:00.521135 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.521102 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" Jul 14 19:01:00.521135 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.521107 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" Jul 14 19:01:00.521232 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.521175 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" Jul 14 19:01:00.521232 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.521184 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa78d8aa-ed1f-4360-ab52-cfce611e5f3f" containerName="istio-proxy" Jul 14 19:01:00.524237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.524212 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.528621 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.528598 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-82771c-openshift-default-dockercfg-cnjb5\"" Jul 14 19:01:00.541871 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.541849 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9"] Jul 14 19:01:00.635587 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635546 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfrp8\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-kube-api-access-xfrp8\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.635587 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635590 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358490a0-fa23-4098-901f-e391e1b09533-istio-podinfo\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635642 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-certs\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635691 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-envoy\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635741 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-istio-token\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635780 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-data\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635801 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-credential-socket\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635821 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358490a0-fa23-4098-901f-e391e1b09533-istiod-ca-cert\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.636121 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.635867 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-socket\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737057 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737023 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-envoy\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737078 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-istio-token\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737102 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-data\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-credential-socket\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358490a0-fa23-4098-901f-e391e1b09533-istiod-ca-cert\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-socket\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737185 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfrp8\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-kube-api-access-xfrp8\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358490a0-fa23-4098-901f-e391e1b09533-istio-podinfo\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737237 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-certs\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737723 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-credential-socket\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737723 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737653 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-socket\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737723 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737696 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-data\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.737874 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.737861 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-certs\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.738174 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.738151 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358490a0-fa23-4098-901f-e391e1b09533-istiod-ca-cert\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.739556 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.739532 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-envoy\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.739911 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.739889 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358490a0-fa23-4098-901f-e391e1b09533-istio-podinfo\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.750250 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.750217 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-istio-token\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.750358 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.750328 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfrp8\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-kube-api-access-xfrp8\") pod \"e2e-api-a-82771c-openshift-default-65466c4695-q5gx9\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.837288 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.837213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:00.978580 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.978547 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9"] Jul 14 19:01:00.982175 ip-10-0-131-199 kubenswrapper[2580]: W0714 19:01:00.982146 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod358490a0_fa23_4098_901f_e391e1b09533.slice/crio-fd9b4a30948cbdc974415f6f261864d6818b95a4afbf3d221f105a045c0e9284 WatchSource:0}: Error finding container fd9b4a30948cbdc974415f6f261864d6818b95a4afbf3d221f105a045c0e9284: Status 404 returned error can't find the container with id fd9b4a30948cbdc974415f6f261864d6818b95a4afbf3d221f105a045c0e9284 Jul 14 19:01:00.984290 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.984251 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:00.984378 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.984338 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:00.984421 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:00.984382 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:01.722241 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:01.722207 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" event={"ID":"358490a0-fa23-4098-901f-e391e1b09533","Type":"ContainerStarted","Data":"ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68"} Jul 14 19:01:01.722698 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:01.722263 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" event={"ID":"358490a0-fa23-4098-901f-e391e1b09533","Type":"ContainerStarted","Data":"fd9b4a30948cbdc974415f6f261864d6818b95a4afbf3d221f105a045c0e9284"} Jul 14 19:01:01.753076 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:01.753029 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podStartSLOduration=1.7530150930000001 podStartE2EDuration="1.753015093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:01.750669135 +0000 UTC m=+2352.475809197" watchObservedRunningTime="2026-07-14 19:01:01.753015093 +0000 UTC m=+2352.478155154" Jul 14 19:01:01.837676 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:01.837636 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:02.842851 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:02.842821 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:03.688184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:03.688147 2580 patch_prober.go:28] interesting pod/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:01:03.688349 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:03.688204 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:01:03.738776 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:03.738713 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:03.739819 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:03.739790 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:04.745393 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.745363 2580 generic.go:356] "Generic (PLEG): container finished" podID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerID="59f448c767fe5d564266d76374b3d0974b58cba5b291c0bf6bd32297407d7e86" exitCode=0 Jul 14 19:01:04.745856 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.745413 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" event={"ID":"bea67678-5d02-4b0f-a5d5-a6688fd3d19f","Type":"ContainerDied","Data":"59f448c767fe5d564266d76374b3d0974b58cba5b291c0bf6bd32297407d7e86"} Jul 14 19:01:04.745856 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.745491 2580 scope.go:117] "RemoveContainer" containerID="16f175cbad20192a3528ebf72a3845fc8c9f48002e9f849594401830188ee417" Jul 14 19:01:04.834200 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.834174 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 19:01:04.871454 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871385 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xj4c\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-kube-api-access-8xj4c\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871454 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871420 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-socket\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871454 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871445 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-credential-socket\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871474 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-data\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871494 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-certs\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871516 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-podinfo\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871533 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-envoy\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871551 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-token\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.871693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871571 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istiod-ca-cert\") pod \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\" (UID: \"bea67678-5d02-4b0f-a5d5-a6688fd3d19f\") " Jul 14 19:01:04.872024 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871691 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:04.872024 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871833 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:04.872024 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.871897 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:04.872440 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.872171 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:01:04.872440 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.872271 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.872440 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.872289 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.872440 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.872304 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.872440 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.872316 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.872440 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.872388 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-data" (OuterVolumeSpecName: "istio-data") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:04.873919 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.873890 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:01:04.874058 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.874039 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-kube-api-access-8xj4c" (OuterVolumeSpecName: "kube-api-access-8xj4c") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "kube-api-access-8xj4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:01:04.874390 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.874374 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:04.874828 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.874806 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-token" (OuterVolumeSpecName: "istio-token") pod "bea67678-5d02-4b0f-a5d5-a6688fd3d19f" (UID: "bea67678-5d02-4b0f-a5d5-a6688fd3d19f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:01:04.973233 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.973197 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.973233 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.973226 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.973233 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.973235 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.973451 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.973243 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:04.973451 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:04.973253 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xj4c\" (UniqueName: \"kubernetes.io/projected/bea67678-5d02-4b0f-a5d5-a6688fd3d19f-kube-api-access-8xj4c\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:05.750888 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:05.750850 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" Jul 14 19:01:05.751357 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:05.750898 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch" event={"ID":"bea67678-5d02-4b0f-a5d5-a6688fd3d19f","Type":"ContainerDied","Data":"324836e1f7f0fb3b762862be00e1290ae9d3fc2765b764a736a227206f4a7bbb"} Jul 14 19:01:05.751357 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:05.750957 2580 scope.go:117] "RemoveContainer" containerID="59f448c767fe5d564266d76374b3d0974b58cba5b291c0bf6bd32297407d7e86" Jul 14 19:01:05.773873 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:05.773848 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch"] Jul 14 19:01:05.783244 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:05.783222 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-08413e-openshift-default-c889b7fc-6mcch"] Jul 14 19:01:07.783309 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:07.783280 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" path="/var/lib/kubelet/pods/bea67678-5d02-4b0f-a5d5-a6688fd3d19f/volumes" Jul 14 19:01:09.129724 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.129656 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9"] Jul 14 19:01:09.130285 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.130261 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" Jul 14 19:01:09.130372 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.130287 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" Jul 14 19:01:09.130426 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.130376 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" Jul 14 19:01:09.130426 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.130387 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" Jul 14 19:01:09.134808 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.134786 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.137339 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.137319 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-601221-openshift-default-dockercfg-bpchs\"" Jul 14 19:01:09.146494 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.146472 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9"] Jul 14 19:01:09.212231 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212198 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-credential-socket\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212231 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212234 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-socket\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212427 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-data\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212427 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212386 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-envoy\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212427 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212419 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-istio-token\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212557 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212458 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-certs\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212557 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212482 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942f7f24-9253-412e-8a2f-5cfb98853436-istio-podinfo\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212557 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212548 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942f7f24-9253-412e-8a2f-5cfb98853436-istiod-ca-cert\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.212673 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.212587 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf56l\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-kube-api-access-gf56l\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.312988 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.312954 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-data\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.312996 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-envoy\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313014 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-istio-token\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313041 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-certs\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313061 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942f7f24-9253-412e-8a2f-5cfb98853436-istio-podinfo\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313121 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942f7f24-9253-412e-8a2f-5cfb98853436-istiod-ca-cert\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313184 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313163 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf56l\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-kube-api-access-gf56l\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313511 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-credential-socket\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313511 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-socket\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313511 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313413 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-data\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313511 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313476 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-certs\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313724 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313548 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-socket\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.313724 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.313576 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-credential-socket\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.314198 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.314176 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942f7f24-9253-412e-8a2f-5cfb98853436-istiod-ca-cert\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.315658 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.315634 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942f7f24-9253-412e-8a2f-5cfb98853436-istio-podinfo\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.315746 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.315695 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-envoy\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.321696 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.321675 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-istio-token\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.321847 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.321790 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf56l\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-kube-api-access-gf56l\") pod \"e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.444998 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.444910 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:09.581859 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.581813 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9"] Jul 14 19:01:09.585523 ip-10-0-131-199 kubenswrapper[2580]: W0714 19:01:09.585497 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942f7f24_9253_412e_8a2f_5cfb98853436.slice/crio-fb7d3ac66d294ff48099f5acb65ac6df7397de1206eb400d80ec4d35d61a06c1 WatchSource:0}: Error finding container fb7d3ac66d294ff48099f5acb65ac6df7397de1206eb400d80ec4d35d61a06c1: Status 404 returned error can't find the container with id fb7d3ac66d294ff48099f5acb65ac6df7397de1206eb400d80ec4d35d61a06c1 Jul 14 19:01:09.587664 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.587635 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:09.587742 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.587702 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:09.587742 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.587729 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:09.771711 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.771682 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" event={"ID":"942f7f24-9253-412e-8a2f-5cfb98853436","Type":"ContainerStarted","Data":"ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041"} Jul 14 19:01:09.771932 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.771738 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" event={"ID":"942f7f24-9253-412e-8a2f-5cfb98853436","Type":"ContainerStarted","Data":"fb7d3ac66d294ff48099f5acb65ac6df7397de1206eb400d80ec4d35d61a06c1"} Jul 14 19:01:09.796731 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:09.796671 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" podStartSLOduration=0.796651514 podStartE2EDuration="796.651514ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:01:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:01:09.794180713 +0000 UTC m=+2360.519320796" watchObservedRunningTime="2026-07-14 19:01:09.796651514 +0000 UTC m=+2360.521791576" Jul 14 19:01:10.445996 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:10.445964 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:10.450855 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:10.450833 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:10.776840 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:10.776812 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:10.777924 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:10.777899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:17.813767 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:17.813736 2580 generic.go:356] "Generic (PLEG): container finished" podID="358490a0-fa23-4098-901f-e391e1b09533" containerID="ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68" exitCode=0 Jul 14 19:01:17.814172 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:17.813819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" event={"ID":"358490a0-fa23-4098-901f-e391e1b09533","Type":"ContainerDied","Data":"ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68"} Jul 14 19:01:17.814216 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:17.814197 2580 scope.go:117] "RemoveContainer" containerID="ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68" Jul 14 19:01:17.815224 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:17.815191 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:17.815316 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:17.815271 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:17.815316 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:17.815309 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:18.739524 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:18.739431 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:18.820437 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:18.820394 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" event={"ID":"358490a0-fa23-4098-901f-e391e1b09533","Type":"ContainerStarted","Data":"5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad"} Jul 14 19:01:18.837504 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:18.837475 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:18.838690 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:18.838659 2580 patch_prober.go:28] interesting pod/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 19:01:18.838842 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:18.838720 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 19:01:19.837702 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:19.837656 2580 patch_prober.go:28] interesting pod/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 19:01:19.838227 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:19.837723 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 19:01:20.839300 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:20.839255 2580 patch_prober.go:28] interesting pod/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 19:01:20.839816 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:20.839322 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 19:01:21.838182 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:21.838138 2580 patch_prober.go:28] interesting pod/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 19:01:21.838387 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:21.838216 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 19:01:22.838787 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:22.838164 2580 patch_prober.go:28] interesting pod/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body= Jul 14 19:01:22.838787 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:22.838251 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" Jul 14 19:01:23.841858 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:23.841824 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:23.842303 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:23.842147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:23.842993 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:23.842973 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:24.863242 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:24.863210 2580 generic.go:356] "Generic (PLEG): container finished" podID="942f7f24-9253-412e-8a2f-5cfb98853436" containerID="ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041" exitCode=0 Jul 14 19:01:24.863676 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:24.863285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" event={"ID":"942f7f24-9253-412e-8a2f-5cfb98853436","Type":"ContainerDied","Data":"ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041"} Jul 14 19:01:24.863841 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:24.863813 2580 scope.go:117] "RemoveContainer" containerID="ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041" Jul 14 19:01:24.864857 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:24.864829 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:24.864928 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:24.864894 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:24.864969 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:24.864926 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:01:25.776951 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:25.776910 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:25.870161 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:25.870122 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" event={"ID":"942f7f24-9253-412e-8a2f-5cfb98853436","Type":"ContainerStarted","Data":"8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6"} Jul 14 19:01:26.445929 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:26.445898 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:26.447163 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:26.447136 2580 patch_prober.go:28] interesting pod/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:01:26.447366 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:26.447342 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 19:01:27.446219 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:27.446172 2580 patch_prober.go:28] interesting pod/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:01:27.446706 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:27.446241 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 19:01:28.445518 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:28.445475 2580 patch_prober.go:28] interesting pod/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:01:28.445731 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:28.445550 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 19:01:29.445977 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:29.445935 2580 patch_prober.go:28] interesting pod/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 14 19:01:29.446585 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:29.446546 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 14 19:01:30.449822 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:30.449792 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:30.450199 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:30.450083 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:30.450855 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:30.450838 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:01:49.782191 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:49.782157 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 19:01:49.785086 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:49.785064 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 19:01:50.490318 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:50.490288 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9"] Jul 14 19:01:50.490556 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:50.490511 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" containerID="cri-o://5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad" gracePeriod=30 Jul 14 19:01:53.844215 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:53.844178 2580 patch_prober.go:28] interesting pod/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 14 19:01:53.844573 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:53.844234 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 19:01:55.743058 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.743026 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:55.831665 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831635 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-data\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.831853 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831675 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358490a0-fa23-4098-901f-e391e1b09533-istiod-ca-cert\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.831853 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831706 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfrp8\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-kube-api-access-xfrp8\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.831853 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831806 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-certs\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.831853 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831828 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-envoy\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.831853 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831850 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-istio-token\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.832125 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358490a0-fa23-4098-901f-e391e1b09533-istio-podinfo\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.832125 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831942 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-credential-socket\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.832125 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.831986 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-socket\") pod \"358490a0-fa23-4098-901f-e391e1b09533\" (UID: \"358490a0-fa23-4098-901f-e391e1b09533\") " Jul 14 19:01:55.832125 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832082 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/358490a0-fa23-4098-901f-e391e1b09533-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:01:55.832125 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832103 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:55.832383 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832289 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:55.832383 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832303 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/358490a0-fa23-4098-901f-e391e1b09533-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.832383 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832324 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.832383 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832361 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:55.832692 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.832665 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-data" (OuterVolumeSpecName: "istio-data") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:55.834267 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.834241 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-istio-token" (OuterVolumeSpecName: "istio-token") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:01:55.834397 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.834376 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/358490a0-fa23-4098-901f-e391e1b09533-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:01:55.834544 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.834526 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:01:55.834585 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.834548 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-kube-api-access-xfrp8" (OuterVolumeSpecName: "kube-api-access-xfrp8") pod "358490a0-fa23-4098-901f-e391e1b09533" (UID: "358490a0-fa23-4098-901f-e391e1b09533"). InnerVolumeSpecName "kube-api-access-xfrp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:01:55.933056 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933032 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.933056 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933051 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.933056 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933059 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/358490a0-fa23-4098-901f-e391e1b09533-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.933230 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933067 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.933230 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933077 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.933230 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933086 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/358490a0-fa23-4098-901f-e391e1b09533-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:55.933230 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:55.933094 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfrp8\" (UniqueName: \"kubernetes.io/projected/358490a0-fa23-4098-901f-e391e1b09533-kube-api-access-xfrp8\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:01:56.006128 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.006098 2580 generic.go:356] "Generic (PLEG): container finished" podID="358490a0-fa23-4098-901f-e391e1b09533" containerID="5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad" exitCode=0 Jul 14 19:01:56.006269 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.006152 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" Jul 14 19:01:56.006269 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.006164 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" event={"ID":"358490a0-fa23-4098-901f-e391e1b09533","Type":"ContainerDied","Data":"5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad"} Jul 14 19:01:56.006269 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.006208 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9" event={"ID":"358490a0-fa23-4098-901f-e391e1b09533","Type":"ContainerDied","Data":"fd9b4a30948cbdc974415f6f261864d6818b95a4afbf3d221f105a045c0e9284"} Jul 14 19:01:56.006269 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.006229 2580 scope.go:117] "RemoveContainer" containerID="5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad" Jul 14 19:01:56.015107 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.015089 2580 scope.go:117] "RemoveContainer" containerID="ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68" Jul 14 19:01:56.022564 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.022547 2580 scope.go:117] "RemoveContainer" containerID="5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad" Jul 14 19:01:56.022844 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:01:56.022821 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad\": container with ID starting with 5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad not found: ID does not exist" containerID="5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad" Jul 14 19:01:56.022940 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.022849 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad"} err="failed to get container status \"5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad\": rpc error: code = NotFound desc = could not find container \"5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad\": container with ID starting with 5b268492e6cd0a27d3b41b30f71c195c208ce0a361987de78d89b08480c5c4ad not found: ID does not exist" Jul 14 19:01:56.022940 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.022867 2580 scope.go:117] "RemoveContainer" containerID="ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68" Jul 14 19:01:56.023127 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:01:56.023111 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68\": container with ID starting with ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68 not found: ID does not exist" containerID="ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68" Jul 14 19:01:56.023164 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.023134 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68"} err="failed to get container status \"ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68\": rpc error: code = NotFound desc = could not find container \"ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68\": container with ID starting with ba7966922290d282feb6e4334df1c140f00f032c129c267f8ff01e207df8dc68 not found: ID does not exist" Jul 14 19:01:56.028936 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.028912 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9"] Jul 14 19:01:56.033434 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:56.033412 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-82771c-openshift-default-65466c4695-q5gx9"] Jul 14 19:01:57.782652 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:01:57.782623 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="358490a0-fa23-4098-901f-e391e1b09533" path="/var/lib/kubelet/pods/358490a0-fa23-4098-901f-e391e1b09533/volumes" Jul 14 19:02:16.714466 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:16.714433 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9"] Jul 14 19:02:16.714900 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:16.714636 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" containerID="cri-o://8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6" gracePeriod=30 Jul 14 19:02:17.664591 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.664553 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb"] Jul 14 19:02:17.665068 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665052 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" Jul 14 19:02:17.665146 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665071 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" Jul 14 19:02:17.665146 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665090 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" Jul 14 19:02:17.665146 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665099 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" Jul 14 19:02:17.665146 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665116 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" Jul 14 19:02:17.665146 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665126 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="bea67678-5d02-4b0f-a5d5-a6688fd3d19f" containerName="istio-proxy" Jul 14 19:02:17.665302 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.665202 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" Jul 14 19:02:17.670560 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.670536 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.675262 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.675185 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-52ef23-openshift-default-dockercfg-kmsxf\"" Jul 14 19:02:17.683943 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.683917 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb"] Jul 14 19:02:17.822726 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.822694 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istiod-ca-cert\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.822747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-certs\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.822786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-envoy\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.822860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-data\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.822984 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-podinfo\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.823028 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-token\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.823096 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-credential-socket\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.823126 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwzfw\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-kube-api-access-vwzfw\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.823213 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.823156 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-socket\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.923848 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.923740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-credential-socket\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.923848 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.923812 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwzfw\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-kube-api-access-vwzfw\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.923865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-socket\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.923905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istiod-ca-cert\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.923969 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-certs\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.923991 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-envoy\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924022 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-data\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924339 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924080 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-podinfo\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924339 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924106 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-token\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924339 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924195 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-credential-socket\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924339 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924270 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-socket\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924885 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924622 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istiod-ca-cert\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924885 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924796 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-certs\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.924885 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.924842 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-data\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.927245 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.927215 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-envoy\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.927457 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.927435 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-podinfo\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.936258 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.936230 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-token\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.936576 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.936554 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwzfw\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-kube-api-access-vwzfw\") pod \"e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:17.985062 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:17.985029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:18.127148 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:18.127117 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb"] Jul 14 19:02:18.129558 ip-10-0-131-199 kubenswrapper[2580]: W0714 19:02:18.129531 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9a06b8f_831c_4ddc_85e8_5a49fbe07272.slice/crio-7246767eb3bb12ed53a6098606e064795c889e4c61cc82d30da704b99738cbf6 WatchSource:0}: Error finding container 7246767eb3bb12ed53a6098606e064795c889e4c61cc82d30da704b99738cbf6: Status 404 returned error can't find the container with id 7246767eb3bb12ed53a6098606e064795c889e4c61cc82d30da704b99738cbf6 Jul 14 19:02:18.131590 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:18.131556 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:18.131659 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:18.131624 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:18.131659 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:18.131655 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:19.112170 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:19.112129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" event={"ID":"f9a06b8f-831c-4ddc-85e8-5a49fbe07272","Type":"ContainerStarted","Data":"55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797"} Jul 14 19:02:19.112648 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:19.112223 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" event={"ID":"f9a06b8f-831c-4ddc-85e8-5a49fbe07272","Type":"ContainerStarted","Data":"7246767eb3bb12ed53a6098606e064795c889e4c61cc82d30da704b99738cbf6"} Jul 14 19:02:19.135371 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:19.135324 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" podStartSLOduration=2.135310404 podStartE2EDuration="2.135310404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:19.133992703 +0000 UTC m=+2429.859132764" watchObservedRunningTime="2026-07-14 19:02:19.135310404 +0000 UTC m=+2429.860450465" Jul 14 19:02:19.986082 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:19.986043 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:19.990970 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:19.990945 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:20.117015 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:20.116987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:20.118069 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:20.118049 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:21.971330 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:21.971300 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:02:22.068985 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.068935 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-istio-token\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.068985 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.068992 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942f7f24-9253-412e-8a2f-5cfb98853436-istio-podinfo\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069251 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069019 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gf56l\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-kube-api-access-gf56l\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069251 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069072 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-certs\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-data\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069282 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942f7f24-9253-412e-8a2f-5cfb98853436-istiod-ca-cert\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069337 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-socket\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069519 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069380 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-envoy\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069519 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069394 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:02:22.069519 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069489 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-credential-socket\") pod \"942f7f24-9253-412e-8a2f-5cfb98853436\" (UID: \"942f7f24-9253-412e-8a2f-5cfb98853436\") " Jul 14 19:02:22.069689 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069662 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:02:22.069958 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069749 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.069958 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069810 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.069958 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069748 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942f7f24-9253-412e-8a2f-5cfb98853436-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:02:22.069958 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.069886 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:02:22.070344 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.070310 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-data" (OuterVolumeSpecName: "istio-data") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:02:22.071538 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.071509 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:02:22.072039 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.072011 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-istio-token" (OuterVolumeSpecName: "istio-token") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:02:22.072109 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.072060 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-kube-api-access-gf56l" (OuterVolumeSpecName: "kube-api-access-gf56l") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "kube-api-access-gf56l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:02:22.072271 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.072254 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/942f7f24-9253-412e-8a2f-5cfb98853436-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "942f7f24-9253-412e-8a2f-5cfb98853436" (UID: "942f7f24-9253-412e-8a2f-5cfb98853436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:02:22.127143 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.127109 2580 generic.go:356] "Generic (PLEG): container finished" podID="942f7f24-9253-412e-8a2f-5cfb98853436" containerID="8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6" exitCode=0 Jul 14 19:02:22.127294 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.127177 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" Jul 14 19:02:22.127294 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.127192 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" event={"ID":"942f7f24-9253-412e-8a2f-5cfb98853436","Type":"ContainerDied","Data":"8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6"} Jul 14 19:02:22.127294 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.127270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9" event={"ID":"942f7f24-9253-412e-8a2f-5cfb98853436","Type":"ContainerDied","Data":"fb7d3ac66d294ff48099f5acb65ac6df7397de1206eb400d80ec4d35d61a06c1"} Jul 14 19:02:22.127294 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.127292 2580 scope.go:117] "RemoveContainer" containerID="8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6" Jul 14 19:02:22.136135 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.136081 2580 scope.go:117] "RemoveContainer" containerID="ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041" Jul 14 19:02:22.146754 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.146727 2580 scope.go:117] "RemoveContainer" containerID="8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6" Jul 14 19:02:22.147057 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:02:22.147035 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6\": container with ID starting with 8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6 not found: ID does not exist" containerID="8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6" Jul 14 19:02:22.147127 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.147072 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6"} err="failed to get container status \"8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6\": rpc error: code = NotFound desc = could not find container \"8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6\": container with ID starting with 8d3683295a0c2481a30ee9f392d8dfd4cfc608e1614c20df8ecda7b0a6e1d9e6 not found: ID does not exist" Jul 14 19:02:22.147127 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.147091 2580 scope.go:117] "RemoveContainer" containerID="ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041" Jul 14 19:02:22.147355 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:02:22.147332 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041\": container with ID starting with ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041 not found: ID does not exist" containerID="ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041" Jul 14 19:02:22.147405 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.147367 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041"} err="failed to get container status \"ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041\": rpc error: code = NotFound desc = could not find container \"ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041\": container with ID starting with ee7125a64662410d06526657a31e42b4a3f9986ceec0cc8a4cbdb669308ac041 not found: ID does not exist" Jul 14 19:02:22.153816 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.153789 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9"] Jul 14 19:02:22.157823 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.157801 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-601221-openshift-default-5b8587dfdf-6f2m9"] Jul 14 19:02:22.170594 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170568 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.170594 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170590 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/942f7f24-9253-412e-8a2f-5cfb98853436-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.170741 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170599 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.170741 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170610 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/942f7f24-9253-412e-8a2f-5cfb98853436-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.170741 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170626 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.170741 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170638 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/942f7f24-9253-412e-8a2f-5cfb98853436-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:22.170741 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:22.170645 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf56l\" (UniqueName: \"kubernetes.io/projected/942f7f24-9253-412e-8a2f-5cfb98853436-kube-api-access-gf56l\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:02:23.784034 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:23.783998 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" path="/var/lib/kubelet/pods/942f7f24-9253-412e-8a2f-5cfb98853436/volumes" Jul 14 19:02:26.334780 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.332985 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r"] Jul 14 19:02:26.335269 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335242 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" Jul 14 19:02:26.335360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335272 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" Jul 14 19:02:26.335360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335286 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" Jul 14 19:02:26.335360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335294 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" Jul 14 19:02:26.335517 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335373 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" Jul 14 19:02:26.335517 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335386 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="358490a0-fa23-4098-901f-e391e1b09533" containerName="istio-proxy" Jul 14 19:02:26.335517 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.335396 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="942f7f24-9253-412e-8a2f-5cfb98853436" containerName="istio-proxy" Jul 14 19:02:26.338432 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.338406 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.340991 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.340970 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-10fa01-openshift-default-dockercfg-h6vmx\"" Jul 14 19:02:26.349951 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.349929 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r"] Jul 14 19:02:26.509669 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509631 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-socket\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.509883 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509678 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-certs\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.509883 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509705 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-envoy\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.509883 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509752 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-credential-socket\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.509883 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509857 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnj7g\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-kube-api-access-vnj7g\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.510103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509892 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-podinfo\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.510103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509919 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7e7821-3dbb-4573-ad30-320746e0f53c-istiod-ca-cert\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.510103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-data\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.510103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.509999 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-token\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611216 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611122 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-socket\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611216 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611176 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-certs\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611220 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-envoy\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611270 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-credential-socket\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611306 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnj7g\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-kube-api-access-vnj7g\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611341 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-podinfo\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611368 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7e7821-3dbb-4573-ad30-320746e0f53c-istiod-ca-cert\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-data\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611459 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611429 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-token\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.611861 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.611591 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-socket\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.612053 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.612030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-certs\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.612202 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.612131 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-data\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.612435 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.612410 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-credential-socket\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.612521 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.612444 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7e7821-3dbb-4573-ad30-320746e0f53c-istiod-ca-cert\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.614516 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.614491 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-envoy\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.615052 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.615030 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-podinfo\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.622371 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.622327 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-token\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.622721 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.622699 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnj7g\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-kube-api-access-vnj7g\") pod \"e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.650574 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.650546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:26.791852 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.791827 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r"] Jul 14 19:02:26.793734 ip-10-0-131-199 kubenswrapper[2580]: W0714 19:02:26.793701 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7e7821_3dbb_4573_ad30_320746e0f53c.slice/crio-564304355b4b3c1fa789d61cb44ede3c70fa2c583f3b30a078c048fcee940cc2 WatchSource:0}: Error finding container 564304355b4b3c1fa789d61cb44ede3c70fa2c583f3b30a078c048fcee940cc2: Status 404 returned error can't find the container with id 564304355b4b3c1fa789d61cb44ede3c70fa2c583f3b30a078c048fcee940cc2 Jul 14 19:02:26.796112 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.796080 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:26.796177 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.796144 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:26.796177 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:26.796172 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:27.157445 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:27.157412 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" event={"ID":"8f7e7821-3dbb-4573-ad30-320746e0f53c","Type":"ContainerStarted","Data":"cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95"} Jul 14 19:02:27.157669 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:27.157461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" event={"ID":"8f7e7821-3dbb-4573-ad30-320746e0f53c","Type":"ContainerStarted","Data":"564304355b4b3c1fa789d61cb44ede3c70fa2c583f3b30a078c048fcee940cc2"} Jul 14 19:02:27.184988 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:27.184937 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podStartSLOduration=1.184922243 podStartE2EDuration="1.184922243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:27.182616857 +0000 UTC m=+2437.907756919" watchObservedRunningTime="2026-07-14 19:02:27.184922243 +0000 UTC m=+2437.910062304" Jul 14 19:02:27.651162 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:27.651111 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:27.656104 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:27.656081 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:28.162252 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:28.162223 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:28.163083 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:28.163064 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:34.199785 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:34.199681 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerID="55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797" exitCode=0 Jul 14 19:02:34.199785 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:34.199747 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" event={"ID":"f9a06b8f-831c-4ddc-85e8-5a49fbe07272","Type":"ContainerDied","Data":"55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797"} Jul 14 19:02:34.200401 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:34.200135 2580 scope.go:117] "RemoveContainer" containerID="55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797" Jul 14 19:02:34.201224 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:34.201195 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:34.201314 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:34.201296 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:34.201356 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:34.201337 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:35.117292 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:35.117263 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:35.206036 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:35.205999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" event={"ID":"f9a06b8f-831c-4ddc-85e8-5a49fbe07272","Type":"ContainerStarted","Data":"03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7"} Jul 14 19:02:35.985712 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:35.985672 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:35.986915 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:35.986885 2580 patch_prober.go:28] interesting pod/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 19:02:35.987027 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:35.986954 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 19:02:36.986278 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:36.986238 2580 patch_prober.go:28] interesting pod/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 19:02:36.986656 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:36.986292 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 19:02:37.986407 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:37.986319 2580 patch_prober.go:28] interesting pod/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 19:02:37.986407 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:37.986374 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 19:02:38.986458 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:38.986410 2580 patch_prober.go:28] interesting pod/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 14 19:02:38.986880 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:38.986486 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 14 19:02:39.989556 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:39.989527 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:39.989957 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:39.989794 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:39.990509 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:39.990490 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:02:41.236514 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:41.236423 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerID="cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95" exitCode=0 Jul 14 19:02:41.236514 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:41.236498 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" event={"ID":"8f7e7821-3dbb-4573-ad30-320746e0f53c","Type":"ContainerDied","Data":"cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95"} Jul 14 19:02:41.237077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:41.237031 2580 scope.go:117] "RemoveContainer" containerID="cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95" Jul 14 19:02:41.238155 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:41.238124 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:41.238223 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:41.238212 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:41.238291 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:41.238276 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 14 19:02:42.243210 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:42.243174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" event={"ID":"8f7e7821-3dbb-4573-ad30-320746e0f53c","Type":"ContainerStarted","Data":"c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c"} Jul 14 19:02:42.651457 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:42.651419 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:42.652533 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:42.652508 2580 patch_prober.go:28] interesting pod/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:02:42.652599 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:42.652570 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 19:02:43.162356 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:43.162322 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:43.651811 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:43.651740 2580 patch_prober.go:28] interesting pod/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:02:43.652187 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:43.651836 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 19:02:44.651264 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:44.651217 2580 patch_prober.go:28] interesting pod/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:02:44.651460 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:44.651299 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 19:02:45.651200 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:45.651163 2580 patch_prober.go:28] interesting pod/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:02:45.651619 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:45.651224 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 19:02:46.651905 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:46.651855 2580 patch_prober.go:28] interesting pod/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Jul 14 19:02:46.652411 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:46.651923 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Jul 14 19:02:47.655027 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:47.654998 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:02:47.656170 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:02:47.656149 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:06:49.811693 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:06:49.811603 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 19:06:49.815542 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:06:49.815513 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9hs2p_bb59207a-e4c3-4621-bf36-45f4c93d71e6/ovn-acl-logging/0.log" Jul 14 19:08:54.990832 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:08:54.990799 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb"] Jul 14 19:08:54.991338 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:08:54.991036 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" containerID="cri-o://03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7" gracePeriod=30 Jul 14 19:09:00.236696 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.236671 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:09:00.388632 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388592 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-credential-socket\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.388632 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388634 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwzfw\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-kube-api-access-vwzfw\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.388938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388657 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-socket\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.388938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388697 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-data\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.388938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388746 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istiod-ca-cert\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.388938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388799 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-podinfo\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.388938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388911 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:00.388938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388930 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-token\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.389253 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.388978 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-envoy\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.389253 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389028 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-certs\") pod \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\" (UID: \"f9a06b8f-831c-4ddc-85e8-5a49fbe07272\") " Jul 14 19:09:00.389253 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389185 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:09:00.389423 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389313 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.389423 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389329 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.389423 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389387 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:00.389423 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389424 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:00.389805 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.389711 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-data" (OuterVolumeSpecName: "istio-data") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:00.391105 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.391080 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-kube-api-access-vwzfw" (OuterVolumeSpecName: "kube-api-access-vwzfw") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "kube-api-access-vwzfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:09:00.391239 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.391181 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-token" (OuterVolumeSpecName: "istio-token") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:09:00.391554 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.391535 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:09:00.391554 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.391541 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f9a06b8f-831c-4ddc-85e8-5a49fbe07272" (UID: "f9a06b8f-831c-4ddc-85e8-5a49fbe07272"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:00.490496 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490468 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.490496 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490491 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.490661 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490503 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.490661 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490514 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwzfw\" (UniqueName: \"kubernetes.io/projected/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-kube-api-access-vwzfw\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.490661 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490531 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.490661 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490543 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.490661 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.490553 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9a06b8f-831c-4ddc-85e8-5a49fbe07272-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:00.960806 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.960749 2580 generic.go:356] "Generic (PLEG): container finished" podID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerID="03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7" exitCode=0 Jul 14 19:09:00.960996 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.960806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" event={"ID":"f9a06b8f-831c-4ddc-85e8-5a49fbe07272","Type":"ContainerDied","Data":"03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7"} Jul 14 19:09:00.960996 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.960835 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" Jul 14 19:09:00.960996 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.960888 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb" event={"ID":"f9a06b8f-831c-4ddc-85e8-5a49fbe07272","Type":"ContainerDied","Data":"7246767eb3bb12ed53a6098606e064795c889e4c61cc82d30da704b99738cbf6"} Jul 14 19:09:00.960996 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.960921 2580 scope.go:117] "RemoveContainer" containerID="03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7" Jul 14 19:09:00.971169 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.971152 2580 scope.go:117] "RemoveContainer" containerID="55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797" Jul 14 19:09:00.978938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.978917 2580 scope.go:117] "RemoveContainer" containerID="03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7" Jul 14 19:09:00.979182 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:09:00.979164 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7\": container with ID starting with 03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7 not found: ID does not exist" containerID="03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7" Jul 14 19:09:00.979231 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.979192 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7"} err="failed to get container status \"03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7\": rpc error: code = NotFound desc = could not find container \"03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7\": container with ID starting with 03955c2db0591b4d33aaa64988719aa7db20de8e3be0a6b8aea78852e37a71a7 not found: ID does not exist" Jul 14 19:09:00.979231 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.979211 2580 scope.go:117] "RemoveContainer" containerID="55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797" Jul 14 19:09:00.979434 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:09:00.979416 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797\": container with ID starting with 55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797 not found: ID does not exist" containerID="55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797" Jul 14 19:09:00.979496 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.979441 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797"} err="failed to get container status \"55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797\": rpc error: code = NotFound desc = could not find container \"55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797\": container with ID starting with 55a219a8943c42f07234152656975b97ef676205c85bc228c6c6cfab8273c797 not found: ID does not exist" Jul 14 19:09:00.985849 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.985821 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb"] Jul 14 19:09:00.989004 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:00.988985 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-52ef23-openshift-default-7dcdf87999-g58vb"] Jul 14 19:09:01.782511 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:01.782481 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" path="/var/lib/kubelet/pods/f9a06b8f-831c-4ddc-85e8-5a49fbe07272/volumes" Jul 14 19:09:21.216141 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:21.216103 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r"] Jul 14 19:09:21.216586 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:21.216363 2580 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" containerID="cri-o://c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c" gracePeriod=30 Jul 14 19:09:26.463705 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.463683 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:09:26.506260 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506194 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-credential-socket\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506260 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506241 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-envoy\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506433 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506276 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-podinfo\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506433 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506300 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vnj7g\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-kube-api-access-vnj7g\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506433 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506334 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-data\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506433 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506376 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-certs\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506433 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506425 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7e7821-3dbb-4573-ad30-320746e0f53c-istiod-ca-cert\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506688 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506451 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-token\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506688 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506480 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-socket\") pod \"8f7e7821-3dbb-4573-ad30-320746e0f53c\" (UID: \"8f7e7821-3dbb-4573-ad30-320746e0f53c\") " Jul 14 19:09:26.506688 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506567 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:26.506885 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506796 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:26.506937 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.506903 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7e7821-3dbb-4573-ad30-320746e0f53c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 19:09:26.507247 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.507168 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:26.507360 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.507343 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-certs\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.507441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.507365 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7e7821-3dbb-4573-ad30-320746e0f53c-istiod-ca-cert\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.507441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.507378 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-workload-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.507441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.507394 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-credential-socket\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.507441 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.507421 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-data" (OuterVolumeSpecName: "istio-data") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:26.508910 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.508888 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 14 19:09:26.509028 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.508981 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-kube-api-access-vnj7g" (OuterVolumeSpecName: "kube-api-access-vnj7g") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "kube-api-access-vnj7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:09:26.509197 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.509176 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:09:26.509293 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.509225 2580 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-token" (OuterVolumeSpecName: "istio-token") pod "8f7e7821-3dbb-4573-ad30-320746e0f53c" (UID: "8f7e7821-3dbb-4573-ad30-320746e0f53c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:09:26.607938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.607916 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnj7g\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-kube-api-access-vnj7g\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.607938 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.607936 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-data\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.608079 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.607945 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-token\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.608079 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.607953 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-envoy\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:26.608079 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:26.607960 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7e7821-3dbb-4573-ad30-320746e0f53c-istio-podinfo\") on node \"ip-10-0-131-199.ec2.internal\" DevicePath \"\"" Jul 14 19:09:27.074692 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.074659 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerID="c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c" exitCode=0 Jul 14 19:09:27.074900 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.074709 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" Jul 14 19:09:27.074900 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.074752 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" event={"ID":"8f7e7821-3dbb-4573-ad30-320746e0f53c","Type":"ContainerDied","Data":"c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c"} Jul 14 19:09:27.074900 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.074836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r" event={"ID":"8f7e7821-3dbb-4573-ad30-320746e0f53c","Type":"ContainerDied","Data":"564304355b4b3c1fa789d61cb44ede3c70fa2c583f3b30a078c048fcee940cc2"} Jul 14 19:09:27.074900 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.074866 2580 scope.go:117] "RemoveContainer" containerID="c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c" Jul 14 19:09:27.086516 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.086496 2580 scope.go:117] "RemoveContainer" containerID="cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95" Jul 14 19:09:27.094502 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.094485 2580 scope.go:117] "RemoveContainer" containerID="c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c" Jul 14 19:09:27.094732 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:09:27.094714 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c\": container with ID starting with c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c not found: ID does not exist" containerID="c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c" Jul 14 19:09:27.094798 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.094740 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c"} err="failed to get container status \"c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c\": rpc error: code = NotFound desc = could not find container \"c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c\": container with ID starting with c41c4ceb6287c46fdc7152497df3ac262f90901360b780ad5f4d457d13c4225c not found: ID does not exist" Jul 14 19:09:27.094798 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.094774 2580 scope.go:117] "RemoveContainer" containerID="cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95" Jul 14 19:09:27.095032 ip-10-0-131-199 kubenswrapper[2580]: E0714 19:09:27.095013 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95\": container with ID starting with cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95 not found: ID does not exist" containerID="cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95" Jul 14 19:09:27.095103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.095038 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95"} err="failed to get container status \"cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95\": rpc error: code = NotFound desc = could not find container \"cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95\": container with ID starting with cc3b2df7d3133480a84c91fa86bd8237fedfae3e2348f2317c6cffddc4447a95 not found: ID does not exist" Jul 14 19:09:27.099549 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.099529 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r"] Jul 14 19:09:27.105886 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.105859 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-10fa01-openshift-default-6f8b58764c-8qd9r"] Jul 14 19:09:27.782914 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.782882 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" path="/var/lib/kubelet/pods/8f7e7821-3dbb-4573-ad30-320746e0f53c/volumes" Jul 14 19:09:27.910263 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.910238 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-2pkvg_7b26258e-25bd-4282-a0da-747ebe60e46d/copy-manifests/0.log" Jul 14 19:09:27.917905 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:27.917879 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-2pkvg_7b26258e-25bd-4282-a0da-747ebe60e46d/manager/0.log" Jul 14 19:09:28.980789 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:28.980743 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-01bc759f-cv4vx_546e61b2-536c-435f-858e-75fe1f9172e3/revoke-keys/0.log" Jul 14 19:09:29.106050 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:29.106023 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-bfd487d2-rzk5h_0229717e-93da-418d-96bd-d653824a02f2/revoke-keys/0.log" Jul 14 19:09:29.529216 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:29.529186 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-a8b422c9-52z5v_8f6347ff-6d5e-43be-bdfc-30845afdcd8a/revoke-keys/0.log" Jul 14 19:09:29.945200 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:29.945117 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-541ed9bd-xkwf5_318f94a0-46fe-455a-97ff-84d9951ec42f/revoke-keys/0.log" Jul 14 19:09:30.061465 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:30.061436 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-34c47f4d-dskpz_62feab2e-4c2e-41fa-ab11-8822302852b7/revoke-keys/0.log" Jul 14 19:09:30.179104 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:30.179075 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-99a89b27-r7tg4_87df4563-c20d-4a37-a406-6cd8fa4dd9f4/revoke-keys/0.log" Jul 14 19:09:32.590580 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:32.590551 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hvndl_538ed20c-81e1-415e-a6f9-6540e81adc1e/manager/0.log" Jul 14 19:09:33.372480 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:33.372449 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qdszj_9001966f-6c5f-4cc3-aae6-557d714ca7c2/discovery/0.log" Jul 14 19:09:33.903311 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:33.903282 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5kgtk_5844b7bc-64bc-4e4c-8d11-07e6f3d310d5/payload-pre-processing/0.log" Jul 14 19:09:34.535157 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:34.535130 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-f96rg_30ea3f47-8521-47bd-95f9-30811d34b616/main/0.log" Jul 14 19:09:34.926161 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:34.926082 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbvgvb7_08dd9b45-aed8-4cb1-be81-696a353071a9/main/0.log" Jul 14 19:09:42.301132 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:42.301052 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-65qbb_f55ab0b2-0a83-4a87-9649-f7e71ffea294/global-pull-secret-syncer/0.log" Jul 14 19:09:42.588611 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:42.588531 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lzkvk_22cb6695-c93f-40b6-8af9-65d3988a2dee/konnectivity-agent/0.log" Jul 14 19:09:42.639844 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:42.639806 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-199.ec2.internal_eaac1d1135f2c4810bad5bfddca8e38b/haproxy/0.log" Jul 14 19:09:48.941993 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:48.941902 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-hvndl_538ed20c-81e1-415e-a6f9-6540e81adc1e/manager/0.log" Jul 14 19:09:51.018502 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.018465 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nf7wx_960c371c-0d49-4558-85cb-3347d8b691d8/node-exporter/0.log" Jul 14 19:09:51.046477 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.046456 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nf7wx_960c371c-0d49-4558-85cb-3347d8b691d8/kube-rbac-proxy/0.log" Jul 14 19:09:51.076985 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.076965 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nf7wx_960c371c-0d49-4558-85cb-3347d8b691d8/init-textfile/0.log" Jul 14 19:09:51.441979 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.441865 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/prometheus/0.log" Jul 14 19:09:51.464799 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.464774 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/config-reloader/0.log" Jul 14 19:09:51.487542 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.487520 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/thanos-sidecar/0.log" Jul 14 19:09:51.509255 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.509227 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/kube-rbac-proxy-web/0.log" Jul 14 19:09:51.535932 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.535909 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/kube-rbac-proxy/0.log" Jul 14 19:09:51.565530 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.565500 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/kube-rbac-proxy-thanos/0.log" Jul 14 19:09:51.591586 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:51.591563 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_39fa3c3a-e65b-47e3-85f4-f653277da65c/init-config-reloader/0.log" Jul 14 19:09:52.770475 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770442 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw"] Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770810 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770825 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770840 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770846 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770857 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770862 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770872 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" Jul 14 19:09:52.770875 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770877 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" Jul 14 19:09:52.771156 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770927 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" Jul 14 19:09:52.771156 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770933 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f7e7821-3dbb-4573-ad30-320746e0f53c" containerName="istio-proxy" Jul 14 19:09:52.771156 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770941 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" Jul 14 19:09:52.771156 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.770947 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9a06b8f-831c-4ddc-85e8-5a49fbe07272" containerName="istio-proxy" Jul 14 19:09:52.775522 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.775505 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:52.778079 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.778055 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26nj6\"/\"kube-root-ca.crt\"" Jul 14 19:09:52.779077 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.779057 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26nj6\"/\"openshift-service-ca.crt\"" Jul 14 19:09:52.779147 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.779063 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26nj6\"/\"default-dockercfg-9br9n\"" Jul 14 19:09:52.783404 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.783382 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw"] Jul 14 19:09:52.912033 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.912006 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-lib-modules\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:52.912155 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.912058 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9j4\" (UniqueName: \"kubernetes.io/projected/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-kube-api-access-5f9j4\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:52.912155 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.912089 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-podres\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:52.912155 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.912130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-proc\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:52.912264 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:52.912166 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-sys\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013223 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013193 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-sys\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013379 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013237 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-lib-modules\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013379 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9j4\" (UniqueName: \"kubernetes.io/projected/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-kube-api-access-5f9j4\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013379 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-podres\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013379 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013314 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-proc\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013379 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013315 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-sys\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013555 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013453 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-lib-modules\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013555 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013464 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-proc\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.013555 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.013475 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-podres\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.021397 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.021341 2580 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9j4\" (UniqueName: \"kubernetes.io/projected/0c7ac5d7-f257-4d2c-bf63-ed68d634673b-kube-api-access-5f9j4\") pod \"perf-node-gather-daemonset-whhjw\" (UID: \"0c7ac5d7-f257-4d2c-bf63-ed68d634673b\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.086582 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.086557 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:53.214868 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:53.214706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw"] Jul 14 19:09:53.217669 ip-10-0-131-199 kubenswrapper[2580]: W0714 19:09:53.217639 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0c7ac5d7_f257_4d2c_bf63_ed68d634673b.slice/crio-36d85ce7df3a23465074b884bdf22a272c666849e27b5cea275d9af27a0a8b87 WatchSource:0}: Error finding container 36d85ce7df3a23465074b884bdf22a272c666849e27b5cea275d9af27a0a8b87: Status 404 returned error can't find the container with id 36d85ce7df3a23465074b884bdf22a272c666849e27b5cea275d9af27a0a8b87 Jul 14 19:09:54.191701 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:54.191663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" event={"ID":"0c7ac5d7-f257-4d2c-bf63-ed68d634673b","Type":"ContainerStarted","Data":"15d0a86443da8f3e485bd45cd21011fdc584d65f1dcb95e0717b57759087854a"} Jul 14 19:09:54.192103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:54.191719 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" event={"ID":"0c7ac5d7-f257-4d2c-bf63-ed68d634673b","Type":"ContainerStarted","Data":"36d85ce7df3a23465074b884bdf22a272c666849e27b5cea275d9af27a0a8b87"} Jul 14 19:09:54.192103 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:54.191741 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:09:54.210817 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:54.210749 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" podStartSLOduration=2.210734432 podStartE2EDuration="2.210734432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:09:54.208589621 +0000 UTC m=+2884.933729681" watchObservedRunningTime="2026-07-14 19:09:54.210734432 +0000 UTC m=+2884.935874493" Jul 14 19:09:55.979928 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:55.979895 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-stjst_81ae0b10-8390-49bd-844c-18be5fb5a09c/dns/0.log" Jul 14 19:09:56.001076 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:56.001051 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-stjst_81ae0b10-8390-49bd-844c-18be5fb5a09c/kube-rbac-proxy/0.log" Jul 14 19:09:56.026345 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:56.026319 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6fzdg_c2b03911-b95b-46cb-bfca-20132d7d34a8/dns-node-resolver/0.log" Jul 14 19:09:56.739900 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:56.739873 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-v9k87_fcbafb7f-bcf0-485d-84f9-83783eef701f/node-ca/0.log" Jul 14 19:09:57.972437 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:57.972403 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qdszj_9001966f-6c5f-4cc3-aae6-557d714ca7c2/discovery/0.log" Jul 14 19:09:58.197469 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:58.197437 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5kgtk_5844b7bc-64bc-4e4c-8d11-07e6f3d310d5/payload-pre-processing/0.log" Jul 14 19:09:59.080617 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:59.080585 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jzc5q_13730500-ea99-40e3-a88a-07a5593ac288/serve-healthcheck-canary/0.log" Jul 14 19:09:59.717045 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:59.717013 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qs8b_85330e72-de6f-4c21-a6b8-a912f561c69d/kube-rbac-proxy/0.log" Jul 14 19:09:59.747782 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:59.747732 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qs8b_85330e72-de6f-4c21-a6b8-a912f561c69d/exporter/0.log" Jul 14 19:09:59.776861 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:09:59.776830 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2qs8b_85330e72-de6f-4c21-a6b8-a912f561c69d/extractor/0.log" Jul 14 19:10:00.206310 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:00.206276 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-whhjw" Jul 14 19:10:03.112664 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:03.112572 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-2pkvg_7b26258e-25bd-4282-a0da-747ebe60e46d/manager/0.log" Jul 14 19:10:03.131191 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:03.131169 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qdszj_9001966f-6c5f-4cc3-aae6-557d714ca7c2/discovery/0.log" Jul 14 19:10:03.137795 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:03.137778 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-2pkvg_7b26258e-25bd-4282-a0da-747ebe60e46d/copy-manifests/0.log" Jul 14 19:10:03.301809 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:03.301777 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5kgtk_5844b7bc-64bc-4e4c-8d11-07e6f3d310d5/payload-pre-processing/0.log" Jul 14 19:10:04.779709 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:04.779678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8c5cc675c-7rv57_9d9b4a24-7d39-4ed5-9607-0eeef4a5399c/manager/0.log" Jul 14 19:10:04.790229 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:04.790192 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-qdszj_9001966f-6c5f-4cc3-aae6-557d714ca7c2/discovery/0.log" Jul 14 19:10:04.938709 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:04.938681 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-5kgtk_5844b7bc-64bc-4e4c-8d11-07e6f3d310d5/payload-pre-processing/0.log" Jul 14 19:10:09.783238 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:09.783208 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-s5g7c_7ed78753-01af-4475-99c0-b9afda552c13/migrator/0.log" Jul 14 19:10:09.802537 ip-10-0-131-199 kubenswrapper[2580]: I0714 19:10:09.802517 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-s5g7c_7ed78753-01af-4475-99c0-b9afda552c13/graceful-termination/0.log"