Jul 05 13:36:35.753466 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 05 13:36:35.753479 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 05 13:36:35.753488 ip-10-0-130-182 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 05 13:36:35.753799 ip-10-0-130-182 systemd[1]: Failed to start Kubernetes Kubelet. Jul 05 13:36:45.287602 ip-10-0-130-182 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 05 13:36:45.287616 ip-10-0-130-182 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 577e516401b04e08b157aa6c3738d993 -- Jul 05 13:38:31.871686 ip-10-0-130-182 systemd[1]: Starting Kubernetes Kubelet... Jul 05 13:38:32.552298 ip-10-0-130-182 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 13:38:32.552298 ip-10-0-130-182 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 05 13:38:32.552298 ip-10-0-130-182 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 13:38:32.552298 ip-10-0-130-182 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 05 13:38:32.552298 ip-10-0-130-182 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 05 13:38:32.553351 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.553222 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 05 13:38:32.559221 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559201 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 13:38:32.559221 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559217 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 13:38:32.559221 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559222 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 13:38:32.559221 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559228 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559233 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559238 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559241 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559246 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559250 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559255 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559261 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559267 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559272 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559277 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559281 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559285 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559289 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559293 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559298 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559310 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559315 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559319 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 13:38:32.559503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559323 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559328 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559332 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559336 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559340 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559345 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559350 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559354 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559358 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559363 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559367 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559372 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559377 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559382 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559388 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559393 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559398 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559404 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559408 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559413 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 13:38:32.560324 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559418 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559423 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559428 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559432 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559438 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559458 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559463 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559467 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559472 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559477 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559481 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559486 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559490 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559495 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559499 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559504 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559508 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559512 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559516 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559521 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 13:38:32.561241 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559526 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559531 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559535 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559540 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559545 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559549 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559565 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559575 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559580 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559586 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559591 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559596 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559601 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559606 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559614 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559620 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559626 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559631 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559635 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 13:38:32.562120 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559641 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559645 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559650 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559654 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559661 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559666 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559670 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559675 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559680 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559686 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559690 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559695 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559700 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559704 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559709 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559713 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559718 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.559723 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561011 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561024 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 13:38:32.563014 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561031 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561040 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561046 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561052 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561057 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561064 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561071 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561079 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561084 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561089 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561093 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561098 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561103 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561107 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561112 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561116 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561121 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561126 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561131 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 13:38:32.563922 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561135 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561140 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561147 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561152 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561156 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561160 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561165 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561170 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561174 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561179 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561183 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561188 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561192 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561198 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561203 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561207 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561212 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561218 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561222 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561228 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 13:38:32.564655 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561232 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561236 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561240 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561244 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561249 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561253 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561258 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561262 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561267 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561271 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561276 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561282 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561287 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561291 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561296 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561301 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561305 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561310 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561314 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561319 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 13:38:32.565276 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561324 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561328 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561333 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561338 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561343 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561348 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561353 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561357 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561362 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561366 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561377 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561382 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561387 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561391 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561396 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561400 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561405 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561410 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561414 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561419 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 13:38:32.565945 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561424 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561428 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561433 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561437 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561459 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561463 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561468 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561473 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561477 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561481 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561486 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561491 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561495 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561500 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561504 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561509 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561513 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.561517 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562186 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 13:38:32.566687 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562195 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562200 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562204 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562210 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562215 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562219 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562224 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562228 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562233 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562237 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562242 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562246 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562251 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562256 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562260 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562264 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562269 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562273 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562278 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562282 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 13:38:32.567209 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562287 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562292 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562296 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562301 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562306 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562310 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562315 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562321 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562328 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562333 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562337 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562342 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562347 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562351 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562356 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562360 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562365 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562371 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562376 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 13:38:32.567833 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562381 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562386 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562393 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562398 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562402 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562407 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562411 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562416 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562421 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562426 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562433 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562459 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562465 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562470 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562474 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562479 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562484 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562489 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562493 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562497 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 13:38:32.568355 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562502 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562507 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562512 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562516 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562521 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562525 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562530 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562534 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562539 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562543 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562549 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562554 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562558 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562565 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562571 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562576 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562580 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562585 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562589 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 13:38:32.569008 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562594 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562599 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562603 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562607 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562612 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562616 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562621 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562625 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562629 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562633 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562640 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562647 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562653 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562659 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562664 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562669 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562673 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562677 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562682 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 13:38:32.569532 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.562686 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562789 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562801 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562812 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562820 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562827 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562832 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562839 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562850 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562856 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562861 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562867 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562873 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562878 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562883 2569 flags.go:64] FLAG: --cgroup-root="" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562889 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562894 2569 flags.go:64] FLAG: --client-ca-file="" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562899 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562904 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562910 2569 flags.go:64] FLAG: --cluster-domain="" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562915 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562920 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562926 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562933 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 05 13:38:32.570055 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562940 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562946 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562951 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562958 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562963 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562968 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562973 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562978 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562984 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562991 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.562996 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563001 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563006 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563010 2569 flags.go:64] FLAG: --enable-server="true" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563016 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563024 2569 flags.go:64] FLAG: --event-burst="100" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563035 2569 flags.go:64] FLAG: --event-qps="50" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563040 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563045 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563051 2569 flags.go:64] FLAG: --eviction-hard="" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563057 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563062 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563068 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563073 2569 flags.go:64] FLAG: --eviction-soft="" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563078 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 05 13:38:32.570702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563083 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563088 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563093 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563098 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563103 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563108 2569 flags.go:64] FLAG: --feature-gates="" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563115 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563120 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563126 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563131 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563137 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563142 2569 flags.go:64] FLAG: --help="false" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563146 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563152 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563157 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563162 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563167 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563173 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563179 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563184 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563188 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563193 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563199 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563207 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 05 13:38:32.571403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563212 2569 flags.go:64] FLAG: --kube-reserved="" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563217 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563222 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563227 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563232 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563237 2569 flags.go:64] FLAG: --lock-file="" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563242 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563247 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563252 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563261 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563267 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563272 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563276 2569 flags.go:64] FLAG: --logging-format="text" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563282 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563288 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563293 2569 flags.go:64] FLAG: --manifest-url="" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563299 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563307 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563312 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563320 2569 flags.go:64] FLAG: --max-pods="110" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563326 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563331 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563335 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563340 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563346 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 05 13:38:32.572073 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563351 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563357 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563369 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563374 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563380 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563385 2569 flags.go:64] FLAG: --pod-cidr="" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563393 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563403 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563408 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563413 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563419 2569 flags.go:64] FLAG: --port="10250" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563424 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563429 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09745d1dfd12e24af" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563435 2569 flags.go:64] FLAG: --qos-reserved="" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563458 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563465 2569 flags.go:64] FLAG: --register-node="true" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563471 2569 flags.go:64] FLAG: --register-with-taints="" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563477 2569 flags.go:64] FLAG: --registry-burst="10" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563482 2569 flags.go:64] FLAG: --registry-qps="5" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563487 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563493 2569 flags.go:64] FLAG: --reserved-memory="" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563499 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563504 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563509 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563514 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 05 13:38:32.572755 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563519 2569 flags.go:64] FLAG: --runonce="false" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563524 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563529 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563534 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563540 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563544 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563550 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563555 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563560 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563565 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563570 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563575 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563580 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563588 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563601 2569 flags.go:64] FLAG: --system-cgroups="" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563607 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563617 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563622 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563627 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563635 2569 flags.go:64] FLAG: --tls-min-version="" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563640 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563645 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563650 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563655 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563661 2569 flags.go:64] FLAG: --v="2" Jul 05 13:38:32.573393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563667 2569 flags.go:64] FLAG: --version="false" Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563674 2569 flags.go:64] FLAG: --vmodule="" Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563680 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.563686 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563878 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563885 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563891 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563896 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563902 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563907 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563912 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563917 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563921 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563926 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563930 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563935 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563940 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563945 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563950 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563954 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 13:38:32.574100 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563961 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563965 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563970 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563975 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563980 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563985 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563989 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563993 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.563998 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564003 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564007 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564011 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564016 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564020 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564025 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564029 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564036 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564041 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564045 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564050 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 13:38:32.574669 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564057 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564062 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564068 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564072 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564077 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564082 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564086 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564090 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564095 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564100 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564105 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564109 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564115 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564120 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564124 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564131 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564135 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564142 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564149 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 13:38:32.575176 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564154 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564159 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564164 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564168 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564172 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564177 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564182 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564186 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564191 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564197 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564202 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564206 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564211 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564215 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564220 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564225 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564231 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564237 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564243 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 13:38:32.575723 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564247 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564252 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564258 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564263 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564267 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564272 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564278 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564283 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564287 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564292 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564297 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564302 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564306 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564311 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564315 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564320 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564324 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564329 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564333 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564338 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 13:38:32.576237 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564343 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564347 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564354 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564359 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.564363 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.564993 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.572719 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.572735 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.572757 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.572763 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572832 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572838 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572842 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572845 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572848 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 13:38:32.576805 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572851 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572854 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572857 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572859 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572862 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572866 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572868 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572878 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572881 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572884 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572886 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572889 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572892 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572895 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572898 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572900 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572903 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572906 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572909 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572912 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 05 13:38:32.577201 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572914 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572917 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572920 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572922 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572926 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572929 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572932 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572935 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572937 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572946 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572950 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572952 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572955 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572958 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572961 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572964 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572966 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572969 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572973 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 13:38:32.577739 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572977 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572987 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572991 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572994 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.572997 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573001 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573004 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573007 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573011 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573014 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573018 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573021 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573025 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573030 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573034 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573036 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573040 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573044 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573047 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 13:38:32.578247 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573050 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573053 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573057 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573059 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573062 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573066 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573069 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573072 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573075 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573078 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573081 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573084 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573086 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573089 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573092 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573100 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573103 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573106 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573109 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573111 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 13:38:32.578756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573114 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573117 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573119 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573122 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573125 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573127 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573130 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573133 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573136 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573139 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573142 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573145 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573148 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573151 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573154 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 13:38:32.579279 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573156 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.573162 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573352 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573357 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573361 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573364 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573367 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573370 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573373 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573377 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573380 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573383 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573386 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573396 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573400 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 05 13:38:32.579684 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573403 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573406 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573409 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573412 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573415 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573417 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573420 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573423 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573426 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573429 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573432 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573435 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573438 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573457 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573461 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573464 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573466 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573469 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573472 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573474 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 05 13:38:32.580131 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573477 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573480 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573483 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573485 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573488 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573491 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573494 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573497 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573500 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573503 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573506 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573515 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573518 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573521 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573523 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573527 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573529 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573532 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573535 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 05 13:38:32.580758 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573537 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573540 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573543 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573546 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573551 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573553 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573557 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573560 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573562 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573565 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573568 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573570 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573573 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573576 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573578 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573581 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573584 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573587 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573589 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 05 13:38:32.581614 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573592 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573596 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573599 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573602 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573605 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573609 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573618 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573622 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573625 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573628 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573631 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573634 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573637 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573640 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573643 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573646 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573649 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573652 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573655 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 05 13:38:32.582244 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573658 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573661 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573664 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573667 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573670 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573672 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573675 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573691 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:32.573695 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.573700 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.573834 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.576105 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.577478 2569 server.go:1019] "Starting client certificate rotation" Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.577617 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 05 13:38:32.582823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.577668 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 05 13:38:32.584029 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.584000 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 05 13:38:32.585408 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.585382 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 05 13:38:32.596562 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.596536 2569 log.go:25] "Validated CRI v1 runtime API" Jul 05 13:38:32.601421 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.601405 2569 log.go:25] "Validated CRI v1 image API" Jul 05 13:38:32.602319 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.602299 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 05 13:38:32.604780 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.604760 2569 fs.go:135] Filesystem UUIDs: map[23fb1a2a-822b-41dd-9d5b-80898018dc66:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d34a9ec5-0e1b-4fa2-9cc7-ad2868a5ba54:/dev/nvme0n1p3] Jul 05 13:38:32.604833 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.604781 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 05 13:38:32.610853 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.610733 2569 manager.go:216] Machine: {Timestamp:2026-07-05 13:38:32.609272609 +0000 UTC m=+0.618742244 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100775 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec265b3130069496fdbc0454bd3db506 SystemUUID:ec265b31-3006-9496-fdbc-0454bd3db506 BootID:577e5164-01b0-4e08-b157-aa6c3738d993 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7d:1a:07:d7:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7d:1a:07:d7:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:43:46:a8:3c:19 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 05 13:38:32.610853 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.610847 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 05 13:38:32.610982 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.610935 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 05 13:38:32.613948 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.613920 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 05 13:38:32.614096 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.613950 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-182.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 05 13:38:32.614688 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.614678 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 05 13:38:32.614723 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.614691 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 05 13:38:32.614723 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.614699 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 05 13:38:32.615330 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.615319 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 05 13:38:32.615818 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.615802 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 05 13:38:32.616898 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.616885 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 05 13:38:32.620958 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.620944 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 05 13:38:32.621141 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621130 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 05 13:38:32.621177 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621165 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 05 13:38:32.621177 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621176 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 05 13:38:32.621249 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621191 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 05 13:38:32.621249 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621201 2569 kubelet.go:398] "Adding apiserver pod source" Jul 05 13:38:32.621249 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621217 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 05 13:38:32.621713 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621684 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 05 13:38:32.621759 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.621731 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 05 13:38:32.626085 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.626061 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 05 13:38:32.628715 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.628699 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 05 13:38:32.628799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.628732 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 05 13:38:32.630082 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630062 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 05 13:38:32.630127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630091 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 05 13:38:32.630127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630103 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 05 13:38:32.630127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630111 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 05 13:38:32.630127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630125 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630148 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630161 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630170 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630182 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630192 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630206 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 05 13:38:32.630254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.630215 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 05 13:38:32.631904 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.631887 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 05 13:38:32.631904 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.631902 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 05 13:38:32.633717 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.633692 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 05 13:38:32.633793 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.633735 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 05 13:38:32.635269 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.635255 2569 server.go:1268] "Started kubelet" Jul 05 13:38:32.635380 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.635349 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 05 13:38:32.635455 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.635364 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 05 13:38:32.635455 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.635427 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 05 13:38:32.636254 ip-10-0-130-182 systemd[1]: Started Kubernetes Kubelet. Jul 05 13:38:32.637250 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.637233 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 05 13:38:32.637870 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.637857 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 05 13:38:32.639668 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.639650 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 05 13:38:32.644694 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.644675 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 05 13:38:32.644769 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.644678 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 05 13:38:32.644820 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.644793 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645683 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645685 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645731 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645830 2569 factory.go:55] Registering systemd factory Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645847 2569 factory.go:223] Registration of the systemd container factory successfully Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645887 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.645897 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646097 2569 factory.go:153] Registering CRI-O factory Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646110 2569 factory.go:223] Registration of the crio container factory successfully Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646170 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646195 2569 factory.go:103] Registering Raw factory Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646211 2569 manager.go:1196] Started watching for new ooms in manager Jul 05 13:38:32.646763 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646730 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-182.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 05 13:38:32.647482 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.646824 2569 manager.go:349] Starting recovery of all containers Jul 05 13:38:32.647712 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.645997 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18bf6859dd3e2f82 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-07-05 13:38:32.635223938 +0000 UTC m=+0.644693553,LastTimestamp:2026-07-05 13:38:32.635223938 +0000 UTC m=+0.644693553,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Jul 05 13:38:32.650980 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.650949 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 05 13:38:32.651115 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.651085 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-182.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 05 13:38:32.661126 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.661103 2569 manager.go:354] Recovery completed Jul 05 13:38:32.665497 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.665361 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.667483 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.667451 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.667483 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.667485 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.667582 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.667497 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.668037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668022 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 05 13:38:32.668037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668037 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 05 13:38:32.668112 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668041 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.668112 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668058 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 05 13:38:32.668794 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668776 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.668849 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668811 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.668849 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.668828 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.670245 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.670228 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-44jfd" Jul 05 13:38:32.670360 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.670347 2569 policy_none.go:49] "None policy: Start" Jul 05 13:38:32.670424 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.670365 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 05 13:38:32.670424 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.670377 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 05 13:38:32.670687 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.670604 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18bf6859df2a4b89 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-182.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-07-05 13:38:32.667474825 +0000 UTC m=+0.676944438,LastTimestamp:2026-07-05 13:38:32.667474825 +0000 UTC m=+0.676944438,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Jul 05 13:38:32.672458 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.672434 2569 policy_none.go:47] "Start" Jul 05 13:38:32.679624 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.679602 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-44jfd" Jul 05 13:38:32.680769 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.680677 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-182.ec2.internal.18bf6859df2a87ad default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-182.ec2.internal,UID:ip-10-0-130-182.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-130-182.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-130-182.ec2.internal,},FirstTimestamp:2026-07-05 13:38:32.667490221 +0000 UTC m=+0.676959835,LastTimestamp:2026-07-05 13:38:32.667490221 +0000 UTC m=+0.676959835,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-182.ec2.internal,}" Jul 05 13:38:32.715144 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715082 2569 manager.go:337] "Starting Device Plugin manager" Jul 05 13:38:32.715144 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.715134 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 05 13:38:32.715324 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715153 2569 server.go:87] "Starting device plugin registration server" Jul 05 13:38:32.715639 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715626 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 05 13:38:32.715687 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715640 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 05 13:38:32.715806 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715768 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 05 13:38:32.715927 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715885 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 05 13:38:32.715927 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.715899 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 05 13:38:32.716510 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.716493 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 05 13:38:32.716589 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.716538 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:32.791307 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.791258 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 05 13:38:32.793487 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.792599 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 05 13:38:32.793487 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.792641 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 05 13:38:32.793487 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.792667 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 05 13:38:32.793487 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.792713 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 05 13:38:32.795515 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.795498 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 13:38:32.816892 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.816869 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.817747 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.817727 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.817839 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.817762 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.817839 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.817774 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.817839 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.817798 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.827599 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.827581 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.827656 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.827603 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-182.ec2.internal\": node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:32.858348 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.858321 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:32.893550 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.893499 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal"] Jul 05 13:38:32.893744 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.893596 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.895260 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.895243 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.895329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.895275 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.895329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.895286 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.897569 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.897556 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.897713 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.897696 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.897776 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.897729 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.898328 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.898310 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.898456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.898344 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.898456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.898358 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.898456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.898391 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.898456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.898419 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.898456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.898430 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.900802 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.900786 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.900864 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.900829 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 05 13:38:32.901548 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.901533 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientMemory" Jul 05 13:38:32.901607 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.901580 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasNoDiskPressure" Jul 05 13:38:32.901607 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.901593 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeHasSufficientPID" Jul 05 13:38:32.923270 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.923245 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.928218 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.928200 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-182.ec2.internal\" not found" node="ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.948561 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.948524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.948561 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.948564 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed7f726f53bfd3508e373fd03ee9ef0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"bed7f726f53bfd3508e373fd03ee9ef0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.948743 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:32.948588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:32.959173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:32.959152 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.049105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.049015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.049105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.049063 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.049105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.049095 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed7f726f53bfd3508e373fd03ee9ef0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"bed7f726f53bfd3508e373fd03ee9ef0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.049312 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.049132 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bed7f726f53bfd3508e373fd03ee9ef0-config\") pod \"kube-apiserver-proxy-ip-10-0-130-182.ec2.internal\" (UID: \"bed7f726f53bfd3508e373fd03ee9ef0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.049312 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.049131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.049312 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.049133 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/29248b30c20510f3cbc8d2bec2a059f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal\" (UID: \"29248b30c20510f3cbc8d2bec2a059f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.059921 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.059892 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.160096 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.160054 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.231734 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.231691 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.231734 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.231731 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.260228 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.260196 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.361093 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.360992 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.462110 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.462084 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.562918 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.562887 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.576537 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.576509 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 05 13:38:33.576673 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.576655 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 05 13:38:33.663700 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:33.663616 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-182.ec2.internal\" not found" Jul 05 13:38:33.682819 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.682780 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-04 13:33:33 +0000 UTC" deadline="2028-02-25 05:22:40.315140156 +0000 UTC" Jul 05 13:38:33.682819 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.682815 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14391h44m6.632329532s" Jul 05 13:38:33.744243 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.744215 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 13:38:33.745192 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.745172 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.760438 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.760408 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 05 13:38:33.762501 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.762483 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" Jul 05 13:38:33.772378 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.772355 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 05 13:38:33.878483 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:33.878449 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29248b30c20510f3cbc8d2bec2a059f1.slice/crio-448b8261ab50633006b4868ccbde381394fcaf364775f5351a02774c5ce3170b WatchSource:0}: Error finding container 448b8261ab50633006b4868ccbde381394fcaf364775f5351a02774c5ce3170b: Status 404 returned error can't find the container with id 448b8261ab50633006b4868ccbde381394fcaf364775f5351a02774c5ce3170b Jul 05 13:38:33.878868 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:33.878839 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed7f726f53bfd3508e373fd03ee9ef0.slice/crio-f873325669067772870e9d6018c424ebe7c0299f761fb75adc377268042f659f WatchSource:0}: Error finding container f873325669067772870e9d6018c424ebe7c0299f761fb75adc377268042f659f: Status 404 returned error can't find the container with id f873325669067772870e9d6018c424ebe7c0299f761fb75adc377268042f659f Jul 05 13:38:33.882211 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.882190 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 13:38:33.892324 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:33.892300 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 13:38:34.054146 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.054049 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 13:38:34.419679 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.419580 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 05 13:38:34.623304 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.623258 2569 apiserver.go:52] "Watching apiserver" Jul 05 13:38:34.633721 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.633670 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 05 13:38:34.634276 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.634191 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-hgg8b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb","openshift-cluster-node-tuning-operator/tuned-99jnj","openshift-dns/node-resolver-8tp7h","openshift-image-registry/node-ca-zdz5r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal","openshift-network-operator/iptables-alerter-765z7","kube-system/konnectivity-agent-vrzwb","kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal","openshift-multus/multus-additional-cni-plugins-7229l","openshift-multus/multus-jq5sl","openshift-multus/network-metrics-daemon-jbfxn","openshift-network-diagnostics/network-check-target-zcct8","openshift-ovn-kubernetes/ovnkube-node-cgzlg"] Jul 05 13:38:34.637710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.637103 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.640070 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.639605 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.642161 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.640664 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 05 13:38:34.642161 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.640931 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-2s2l9\"" Jul 05 13:38:34.643710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.642999 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zm276\"" Jul 05 13:38:34.643710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.643300 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.644649 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.644101 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.644649 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.644270 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 05 13:38:34.645554 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.645005 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:34.645554 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.645102 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:34.645554 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.645263 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 05 13:38:34.645554 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.645324 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.645554 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.645467 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 05 13:38:34.647554 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.647541 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.649417 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.649361 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 05 13:38:34.650008 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.649635 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.650008 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.649690 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 05 13:38:34.650008 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.649924 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 05 13:38:34.650194 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.650152 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-z2k5t\"" Jul 05 13:38:34.650271 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.650252 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 05 13:38:34.652650 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.652613 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.652998 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.652976 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 05 13:38:34.653307 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.653283 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p279v\"" Jul 05 13:38:34.653579 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.653559 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.654578 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.654552 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.656096 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.654778 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.656096 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.654805 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.656096 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.654873 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:34.656096 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.655748 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657480 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-sys-fs\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-log-socket\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657544 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-host\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657569 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-serviceca\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f72a14a-0f23-403a-8410-3085f801802e-konnectivity-ca\") pod \"konnectivity-agent-vrzwb\" (UID: \"6f72a14a-0f23-403a-8410-3085f801802e\") " pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-socket-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-var-lib-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657703 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-ovnkube-config\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9694b03a-00a4-4b05-a482-fb25462145f1-ovn-node-metrics-cert\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657752 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-registration-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657775 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-etc-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-node-log\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-cni-bin\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657905 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xzhh\" (UniqueName: \"kubernetes.io/projected/9694b03a-00a4-4b05-a482-fb25462145f1-kube-api-access-6xzhh\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.658393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-env-overrides\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.657976 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-kubelet\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658002 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-systemd-units\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658014 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-vbkcn\"" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658048 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658080 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-ovn\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658174 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-cni-netd\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658202 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-run-netns\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658263 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-ovnkube-script-lib\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658372 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f72a14a-0f23-403a-8410-3085f801802e-agent-certs\") pod \"konnectivity-agent-vrzwb\" (UID: \"6f72a14a-0f23-403a-8410-3085f801802e\") " pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658399 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-device-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658424 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658465 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdpjr\" (UniqueName: \"kubernetes.io/projected/a5df9764-35a8-4d25-9354-dccd243658cc-kube-api-access-vdpjr\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-systemd\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658547 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.659467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658537 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d28rn\" (UniqueName: \"kubernetes.io/projected/1d342df4-57dc-415e-9232-a44b96de4d37-kube-api-access-d28rn\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvmc2\" (UniqueName: \"kubernetes.io/projected/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-kube-api-access-qvmc2\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658883 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658913 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-slash\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.658940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.659721 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.660150 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.660472 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.660349 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.660833 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.661349 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rv8pc\"" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.661594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.661708 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.661987 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.662039 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vdl8k\"" Jul 05 13:38:34.662456 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.662292 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 05 13:38:34.663849 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.663670 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 05 13:38:34.663849 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.663819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:34.664002 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.663890 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:34.664970 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.664468 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.664970 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.664684 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wg27n\"" Jul 05 13:38:34.664970 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.664823 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.667944 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.667408 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.670379 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.670324 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 05 13:38:34.670698 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.670677 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xkdhb\"" Jul 05 13:38:34.670800 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.670687 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 05 13:38:34.685121 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.685093 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-28hjx" Jul 05 13:38:34.693975 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.693950 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-28hjx" Jul 05 13:38:34.747014 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.746981 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 05 13:38:34.759476 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:34.759645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759485 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-system-cni-dir\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.759645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kq5x\" (UniqueName: \"kubernetes.io/projected/a98320c7-b668-4c1f-bfd6-7afe069bb368-kube-api-access-5kq5x\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.759645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759570 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-socket-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.759921 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.759658 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:34.759921 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759688 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-var-lib-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.759921 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.759750 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:35.259710005 +0000 UTC m=+3.269179620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:34.759921 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759838 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-socket-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.759921 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759868 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2cb94bd-1a62-46db-b988-8726f95cea99-kubelet-config\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759897 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-var-lib-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.759987 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-kubelet\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760020 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-conf-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-modprobe-d\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760056 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-etc-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760092 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-node-log\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760149 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-node-log\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-cni-bin\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760172 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-etc-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xzhh\" (UniqueName: \"kubernetes.io/projected/9694b03a-00a4-4b05-a482-fb25462145f1-kube-api-access-6xzhh\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760226 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bbf35c9-8555-4999-8b67-38431ab8d634-host-slash\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760249 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-cni-multus\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760263 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-cni-bin\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760328 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bbf35c9-8555-4999-8b67-38431ab8d634-iptables-alerter-script\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-os-release\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-socket-dir-parent\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-k8s-cni-cncf-io\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a3c6df-c47e-4738-a4fd-01179772eb73-tmp-dir\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-ovn\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760628 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-var-lib-kubelet\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760693 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-ovn\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-ovnkube-script-lib\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-cnibin\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.760834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760789 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-cni-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760805 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-run-ovn-kubernetes\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760827 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-hostroot\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-device-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760919 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-device-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760907 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760959 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.760985 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d28rn\" (UniqueName: \"kubernetes.io/projected/1d342df4-57dc-415e-9232-a44b96de4d37-kube-api-access-d28rn\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761013 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvmc2\" (UniqueName: \"kubernetes.io/projected/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-kube-api-access-qvmc2\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f72a14a-0f23-403a-8410-3085f801802e-konnectivity-ca\") pod \"konnectivity-agent-vrzwb\" (UID: \"6f72a14a-0f23-403a-8410-3085f801802e\") " pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761042 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-etc-selinux\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761039 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-openvswitch\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761081 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761203 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761356 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.761716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761515 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-ovnkube-script-lib\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761203 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761771 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f72a14a-0f23-403a-8410-3085f801802e-konnectivity-ca\") pod \"konnectivity-agent-vrzwb\" (UID: \"6f72a14a-0f23-403a-8410-3085f801802e\") " pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761777 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-sys-fs\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761832 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-sys-fs\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-log-socket\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-host\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761932 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-serviceca\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761936 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-log-socket\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761976 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-host\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.761962 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdmxz\" (UniqueName: \"kubernetes.io/projected/e7f25710-f553-4569-8644-826c6adb6727-kube-api-access-pdmxz\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-system-cni-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762078 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-cni-bin\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysctl-conf\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762149 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-ovnkube-config\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-multus-certs\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762334 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:34.762473 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762391 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysconfig\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-sys\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9694b03a-00a4-4b05-a482-fb25462145f1-ovn-node-metrics-cert\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-registration-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82a3c6df-c47e-4738-a4fd-01179772eb73-hosts-file\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762490 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-serviceca\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762505 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-kubernetes\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762554 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-lib-modules\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762601 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/a5df9764-35a8-4d25-9354-dccd243658cc-registration-dir\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762604 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-env-overrides\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-kubelet\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-systemd-units\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2cb94bd-1a62-46db-b988-8726f95cea99-dbus\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762754 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-os-release\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762779 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trlmr\" (UniqueName: \"kubernetes.io/projected/b8f07c2e-c429-43df-840a-b38476bd199c-kube-api-access-trlmr\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdxst\" (UniqueName: \"kubernetes.io/projected/5bbf35c9-8555-4999-8b67-38431ab8d634-kube-api-access-tdxst\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762880 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e7f25710-f553-4569-8644-826c6adb6727-multus-daemon-config\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.763409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.762923 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-ovnkube-config\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-kubelet\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763032 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9694b03a-00a4-4b05-a482-fb25462145f1-env-overrides\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-cni-netd\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763128 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-cni-netd\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763150 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7f25710-f553-4569-8644-826c6adb6727-cni-binary-copy\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763156 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-systemd-units\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763199 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-netns\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763213 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-etc-kubernetes\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6ws5\" (UniqueName: \"kubernetes.io/projected/82a3c6df-c47e-4738-a4fd-01179772eb73-kube-api-access-b6ws5\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763322 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysctl-d\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-host\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763464 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-tuned\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-run-netns\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763536 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f72a14a-0f23-403a-8410-3085f801802e-agent-certs\") pod \"konnectivity-agent-vrzwb\" (UID: \"6f72a14a-0f23-403a-8410-3085f801802e\") " pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763581 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763589 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-run-netns\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.764185 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-systemd\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-run\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.763679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-cnibin\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdpjr\" (UniqueName: \"kubernetes.io/projected/a5df9764-35a8-4d25-9354-dccd243658cc-kube-api-access-vdpjr\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-systemd\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764463 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764539 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-slash\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a98320c7-b668-4c1f-bfd6-7afe069bb368-tmp\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764774 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-host-slash\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.765003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.764773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9694b03a-00a4-4b05-a482-fb25462145f1-run-systemd\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.768092 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.767668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9694b03a-00a4-4b05-a482-fb25462145f1-ovn-node-metrics-cert\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.768092 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.767786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f72a14a-0f23-403a-8410-3085f801802e-agent-certs\") pod \"konnectivity-agent-vrzwb\" (UID: \"6f72a14a-0f23-403a-8410-3085f801802e\") " pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.772488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.772408 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xzhh\" (UniqueName: \"kubernetes.io/projected/9694b03a-00a4-4b05-a482-fb25462145f1-kube-api-access-6xzhh\") pod \"ovnkube-node-cgzlg\" (UID: \"9694b03a-00a4-4b05-a482-fb25462145f1\") " pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.773496 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.773474 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d28rn\" (UniqueName: \"kubernetes.io/projected/1d342df4-57dc-415e-9232-a44b96de4d37-kube-api-access-d28rn\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:34.773673 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.773538 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvmc2\" (UniqueName: \"kubernetes.io/projected/cce826b0-2bf5-4efc-a2dc-ef404b5449cf-kube-api-access-qvmc2\") pod \"node-ca-zdz5r\" (UID: \"cce826b0-2bf5-4efc-a2dc-ef404b5449cf\") " pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.774455 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.774389 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdpjr\" (UniqueName: \"kubernetes.io/projected/a5df9764-35a8-4d25-9354-dccd243658cc-kube-api-access-vdpjr\") pod \"aws-ebs-csi-driver-node-wn4tb\" (UID: \"a5df9764-35a8-4d25-9354-dccd243658cc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.798380 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.798314 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"29248b30c20510f3cbc8d2bec2a059f1","Type":"ContainerStarted","Data":"448b8261ab50633006b4868ccbde381394fcaf364775f5351a02774c5ce3170b"} Jul 05 13:38:34.799676 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.799644 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"bed7f726f53bfd3508e373fd03ee9ef0","Type":"ContainerStarted","Data":"f873325669067772870e9d6018c424ebe7c0299f761fb75adc377268042f659f"} Jul 05 13:38:34.865041 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.864978 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2cb94bd-1a62-46db-b988-8726f95cea99-dbus\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.865041 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-os-release\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865057 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trlmr\" (UniqueName: \"kubernetes.io/projected/b8f07c2e-c429-43df-840a-b38476bd199c-kube-api-access-trlmr\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdxst\" (UniqueName: \"kubernetes.io/projected/5bbf35c9-8555-4999-8b67-38431ab8d634-kube-api-access-tdxst\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865108 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e7f25710-f553-4569-8644-826c6adb6727-multus-daemon-config\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865130 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7f25710-f553-4569-8644-826c6adb6727-cni-binary-copy\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-netns\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-os-release\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-etc-kubernetes\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6ws5\" (UniqueName: \"kubernetes.io/projected/82a3c6df-c47e-4738-a4fd-01179772eb73-kube-api-access-b6ws5\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysctl-d\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-host\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865280 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f2cb94bd-1a62-46db-b988-8726f95cea99-dbus\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.865305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865295 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-tuned\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865348 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-systemd\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-run\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-cnibin\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865439 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a98320c7-b668-4c1f-bfd6-7afe069bb368-tmp\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865550 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-netns\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-system-cni-dir\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kq5x\" (UniqueName: \"kubernetes.io/projected/a98320c7-b668-4c1f-bfd6-7afe069bb368-kube-api-access-5kq5x\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865617 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-system-cni-dir\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865641 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2cb94bd-1a62-46db-b988-8726f95cea99-kubelet-config\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-kubelet\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865701 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-conf-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865727 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-modprobe-d\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865758 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bbf35c9-8555-4999-8b67-38431ab8d634-host-slash\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-cni-multus\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.865991 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865843 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865219 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-etc-kubernetes\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bbf35c9-8555-4999-8b67-38431ab8d634-iptables-alerter-script\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865924 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e7f25710-f553-4569-8644-826c6adb6727-cni-binary-copy\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-os-release\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.865976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-socket-dir-parent\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866004 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-k8s-cni-cncf-io\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a3c6df-c47e-4738-a4fd-01179772eb73-tmp-dir\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866067 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866105 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-var-lib-kubelet\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866140 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-cnibin\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-cni-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-hostroot\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdmxz\" (UniqueName: \"kubernetes.io/projected/e7f25710-f553-4569-8644-826c6adb6727-kube-api-access-pdmxz\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-system-cni-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866296 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-cni-bin\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysctl-conf\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.866866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866351 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-multus-certs\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866379 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysconfig\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e7f25710-f553-4569-8644-826c6adb6727-multus-daemon-config\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866490 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5bbf35c9-8555-4999-8b67-38431ab8d634-iptables-alerter-script\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866552 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-modprobe-d\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866564 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5bbf35c9-8555-4999-8b67-38431ab8d634-host-slash\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866588 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-sys\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866610 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-cni-multus\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866621 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82a3c6df-c47e-4738-a4fd-01179772eb73-hosts-file\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866674 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/82a3c6df-c47e-4738-a4fd-01179772eb73-hosts-file\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.866703 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.866764 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:35.366745375 +0000 UTC m=+3.376214990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866779 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysctl-d\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866918 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f2cb94bd-1a62-46db-b988-8726f95cea99-kubelet-config\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.866969 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-kubelet\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867017 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-conf-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.867844 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-sys\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867143 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-systemd\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867188 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-system-cni-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867215 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-multus-certs\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-run-k8s-cni-cncf-io\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867288 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-os-release\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867316 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-cni-binary-copy\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867361 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-socket-dir-parent\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867388 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-host\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867423 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-run\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867488 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-kubernetes\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867493 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-cnibin\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-lib-modules\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867521 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysctl-conf\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867557 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-sysconfig\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867562 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-host-var-lib-cni-bin\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867564 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/82a3c6df-c47e-4738-a4fd-01179772eb73-tmp-dir\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.868731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867608 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-kubernetes\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867681 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-lib-modules\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867681 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-hostroot\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867676 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a98320c7-b668-4c1f-bfd6-7afe069bb368-var-lib-kubelet\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867712 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867732 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e7f25710-f553-4569-8644-826c6adb6727-multus-cni-dir\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.867770 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8f07c2e-c429-43df-840a-b38476bd199c-cnibin\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.869618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.868021 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8f07c2e-c429-43df-840a-b38476bd199c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.869843 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.869638 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a98320c7-b668-4c1f-bfd6-7afe069bb368-etc-tuned\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.870087 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.870066 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a98320c7-b668-4c1f-bfd6-7afe069bb368-tmp\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.877707 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.877668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdxst\" (UniqueName: \"kubernetes.io/projected/5bbf35c9-8555-4999-8b67-38431ab8d634-kube-api-access-tdxst\") pod \"iptables-alerter-765z7\" (UID: \"5bbf35c9-8555-4999-8b67-38431ab8d634\") " pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:34.877707 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.877715 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:38:34.878009 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.877741 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:38:34.878009 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.877757 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:34.878009 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:34.877840 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:38:35.377820773 +0000 UTC m=+3.387290399 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:34.880072 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.880002 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kq5x\" (UniqueName: \"kubernetes.io/projected/a98320c7-b668-4c1f-bfd6-7afe069bb368-kube-api-access-5kq5x\") pod \"tuned-99jnj\" (UID: \"a98320c7-b668-4c1f-bfd6-7afe069bb368\") " pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:34.880293 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.880272 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdmxz\" (UniqueName: \"kubernetes.io/projected/e7f25710-f553-4569-8644-826c6adb6727-kube-api-access-pdmxz\") pod \"multus-jq5sl\" (UID: \"e7f25710-f553-4569-8644-826c6adb6727\") " pod="openshift-multus/multus-jq5sl" Jul 05 13:38:34.880960 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.880942 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trlmr\" (UniqueName: \"kubernetes.io/projected/b8f07c2e-c429-43df-840a-b38476bd199c-kube-api-access-trlmr\") pod \"multus-additional-cni-plugins-7229l\" (UID: \"b8f07c2e-c429-43df-840a-b38476bd199c\") " pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:34.881619 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.881590 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6ws5\" (UniqueName: \"kubernetes.io/projected/82a3c6df-c47e-4738-a4fd-01179772eb73-kube-api-access-b6ws5\") pod \"node-resolver-8tp7h\" (UID: \"82a3c6df-c47e-4738-a4fd-01179772eb73\") " pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:34.954122 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.954020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:34.965574 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.965534 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" Jul 05 13:38:34.982474 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.982442 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:38:34.983520 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.983497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zdz5r" Jul 05 13:38:34.993475 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:34.993454 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-765z7" Jul 05 13:38:35.002464 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.002445 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7229l" Jul 05 13:38:35.019323 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.019298 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jq5sl" Jul 05 13:38:35.027393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.027361 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8tp7h" Jul 05 13:38:35.034415 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.034398 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-99jnj" Jul 05 13:38:35.271646 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.271544 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:35.271816 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.271739 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:35.271893 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.271829 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:36.271805715 +0000 UTC m=+4.281275328 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:35.372816 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.372776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:35.373026 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.372942 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:35.373026 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.373016 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:36.373000339 +0000 UTC m=+4.382469939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:35.473943 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.473903 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:35.474109 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.474087 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:38:35.474149 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.474110 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:38:35.474149 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.474126 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:35.474216 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.474179 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:38:36.474164538 +0000 UTC m=+4.483634137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:35.697767 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.697683 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-04 13:33:35 +0000 UTC" deadline="2028-03-02 14:06:54.048334735 +0000 UTC" Jul 05 13:38:35.697767 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.697718 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14544h28m18.350620345s" Jul 05 13:38:35.792886 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:35.792850 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:35.793081 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:35.792991 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:36.073771 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.073734 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbf35c9_8555_4999_8b67_38431ab8d634.slice/crio-38070de645216d10b6c9c01889876ea29e0a0b4142d7937c8b37a83ccc79677d WatchSource:0}: Error finding container 38070de645216d10b6c9c01889876ea29e0a0b4142d7937c8b37a83ccc79677d: Status 404 returned error can't find the container with id 38070de645216d10b6c9c01889876ea29e0a0b4142d7937c8b37a83ccc79677d Jul 05 13:38:36.076439 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.076410 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98320c7_b668_4c1f_bfd6_7afe069bb368.slice/crio-cc362c7b6acbc5923f7a21f3c24e59dacb4c80ecbb2d144c24c26d82d42e2769 WatchSource:0}: Error finding container cc362c7b6acbc5923f7a21f3c24e59dacb4c80ecbb2d144c24c26d82d42e2769: Status 404 returned error can't find the container with id cc362c7b6acbc5923f7a21f3c24e59dacb4c80ecbb2d144c24c26d82d42e2769 Jul 05 13:38:36.079903 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.079877 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5df9764_35a8_4d25_9354_dccd243658cc.slice/crio-efe8265f76dabc5ee2cd7f20b7f351ff961e03fd7ffcbe52897a7afd5da86a47 WatchSource:0}: Error finding container efe8265f76dabc5ee2cd7f20b7f351ff961e03fd7ffcbe52897a7afd5da86a47: Status 404 returned error can't find the container with id efe8265f76dabc5ee2cd7f20b7f351ff961e03fd7ffcbe52897a7afd5da86a47 Jul 05 13:38:36.081439 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.081344 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce826b0_2bf5_4efc_a2dc_ef404b5449cf.slice/crio-0957c31adb7d4cd0384d7097c08f56f7c896c0c23a4e58417511c092680c0e2a WatchSource:0}: Error finding container 0957c31adb7d4cd0384d7097c08f56f7c896c0c23a4e58417511c092680c0e2a: Status 404 returned error can't find the container with id 0957c31adb7d4cd0384d7097c08f56f7c896c0c23a4e58417511c092680c0e2a Jul 05 13:38:36.082379 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.082354 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9694b03a_00a4_4b05_a482_fb25462145f1.slice/crio-2a3ab338eb2e32db4a8129baec0145eea3650a12592cdcbc9c8b3f35daf5f7f9 WatchSource:0}: Error finding container 2a3ab338eb2e32db4a8129baec0145eea3650a12592cdcbc9c8b3f35daf5f7f9: Status 404 returned error can't find the container with id 2a3ab338eb2e32db4a8129baec0145eea3650a12592cdcbc9c8b3f35daf5f7f9 Jul 05 13:38:36.084116 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.084056 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82a3c6df_c47e_4738_a4fd_01179772eb73.slice/crio-20cf1d5c16236d92859d497c98f5238432882f205e3bd3cde6a26731bdb95cba WatchSource:0}: Error finding container 20cf1d5c16236d92859d497c98f5238432882f205e3bd3cde6a26731bdb95cba: Status 404 returned error can't find the container with id 20cf1d5c16236d92859d497c98f5238432882f205e3bd3cde6a26731bdb95cba Jul 05 13:38:36.086048 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.085853 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f25710_f553_4569_8644_826c6adb6727.slice/crio-17e9ef0388fe3a82ed64d2472ec3ce93e10c7c7d030fd9eb5461bf5801e4627a WatchSource:0}: Error finding container 17e9ef0388fe3a82ed64d2472ec3ce93e10c7c7d030fd9eb5461bf5801e4627a: Status 404 returned error can't find the container with id 17e9ef0388fe3a82ed64d2472ec3ce93e10c7c7d030fd9eb5461bf5801e4627a Jul 05 13:38:36.086857 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.086834 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f72a14a_0f23_403a_8410_3085f801802e.slice/crio-c10ea681515e6cbcbbce687b862c009b6acd02a2b8e70a85dc6aefb49c021e9f WatchSource:0}: Error finding container c10ea681515e6cbcbbce687b862c009b6acd02a2b8e70a85dc6aefb49c021e9f: Status 404 returned error can't find the container with id c10ea681515e6cbcbbce687b862c009b6acd02a2b8e70a85dc6aefb49c021e9f Jul 05 13:38:36.089469 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:38:36.089352 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f07c2e_c429_43df_840a_b38476bd199c.slice/crio-464a305831bd222fec8fe61396a16d342716dd34ebab7a83a9b7339482b66556 WatchSource:0}: Error finding container 464a305831bd222fec8fe61396a16d342716dd34ebab7a83a9b7339482b66556: Status 404 returned error can't find the container with id 464a305831bd222fec8fe61396a16d342716dd34ebab7a83a9b7339482b66556 Jul 05 13:38:36.280634 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.280400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:36.280815 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.280556 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:36.280815 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.280774 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:38.280750956 +0000 UTC m=+6.290220576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:36.381551 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.381518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:36.381679 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.381663 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:36.381731 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.381722 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:38.381708727 +0000 UTC m=+6.391178326 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:36.483436 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.483357 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:36.483584 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.483535 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:38:36.483584 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.483557 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:38:36.483584 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.483571 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:36.483778 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.483632 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:38:38.483617225 +0000 UTC m=+6.493086842 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:36.699001 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.698902 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-04 13:33:35 +0000 UTC" deadline="2027-12-15 05:03:27.45916026 +0000 UTC" Jul 05 13:38:36.699001 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.698974 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12663h24m50.760190366s" Jul 05 13:38:36.800308 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.800278 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:36.800492 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.800426 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:36.800896 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.800875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:36.801043 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:36.800998 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:36.841090 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.840952 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-765z7" event={"ID":"5bbf35c9-8555-4999-8b67-38431ab8d634","Type":"ContainerStarted","Data":"38070de645216d10b6c9c01889876ea29e0a0b4142d7937c8b37a83ccc79677d"} Jul 05 13:38:36.851494 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.850919 2569 generic.go:356] "Generic (PLEG): container finished" podID="29248b30c20510f3cbc8d2bec2a059f1" containerID="31d45139b5f8dadd153dc8d55c88276fa3a876cc7dce235ee9c4b03330f08316" exitCode=0 Jul 05 13:38:36.851494 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.851022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"29248b30c20510f3cbc8d2bec2a059f1","Type":"ContainerDied","Data":"31d45139b5f8dadd153dc8d55c88276fa3a876cc7dce235ee9c4b03330f08316"} Jul 05 13:38:36.865843 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.865811 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" event={"ID":"bed7f726f53bfd3508e373fd03ee9ef0","Type":"ContainerStarted","Data":"b5f842ee6e083128f6cb950f9700e3d1d1a59c73a60d48a10ea8ccf8e0e3c32d"} Jul 05 13:38:36.879626 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.879547 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zdz5r" event={"ID":"cce826b0-2bf5-4efc-a2dc-ef404b5449cf","Type":"ContainerStarted","Data":"0957c31adb7d4cd0384d7097c08f56f7c896c0c23a4e58417511c092680c0e2a"} Jul 05 13:38:36.883342 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.883272 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jq5sl" event={"ID":"e7f25710-f553-4569-8644-826c6adb6727","Type":"ContainerStarted","Data":"17e9ef0388fe3a82ed64d2472ec3ce93e10c7c7d030fd9eb5461bf5801e4627a"} Jul 05 13:38:36.889292 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.889231 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vrzwb" event={"ID":"6f72a14a-0f23-403a-8410-3085f801802e","Type":"ContainerStarted","Data":"c10ea681515e6cbcbbce687b862c009b6acd02a2b8e70a85dc6aefb49c021e9f"} Jul 05 13:38:36.900506 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.900410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8tp7h" event={"ID":"82a3c6df-c47e-4738-a4fd-01179772eb73","Type":"ContainerStarted","Data":"20cf1d5c16236d92859d497c98f5238432882f205e3bd3cde6a26731bdb95cba"} Jul 05 13:38:36.914160 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.914121 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"2a3ab338eb2e32db4a8129baec0145eea3650a12592cdcbc9c8b3f35daf5f7f9"} Jul 05 13:38:36.939909 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.939829 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerStarted","Data":"464a305831bd222fec8fe61396a16d342716dd34ebab7a83a9b7339482b66556"} Jul 05 13:38:36.962544 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.962494 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" event={"ID":"a5df9764-35a8-4d25-9354-dccd243658cc","Type":"ContainerStarted","Data":"efe8265f76dabc5ee2cd7f20b7f351ff961e03fd7ffcbe52897a7afd5da86a47"} Jul 05 13:38:36.992788 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:36.992714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99jnj" event={"ID":"a98320c7-b668-4c1f-bfd6-7afe069bb368","Type":"ContainerStarted","Data":"cc362c7b6acbc5923f7a21f3c24e59dacb4c80ecbb2d144c24c26d82d42e2769"} Jul 05 13:38:37.793517 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:37.793478 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:37.794061 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:37.793614 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:38.017739 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.017702 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" event={"ID":"29248b30c20510f3cbc8d2bec2a059f1","Type":"ContainerStarted","Data":"fe4113fa9372742c9de1f665ddfa0bf547c435d5ef9d4ee6bbaa7d8665e11d9e"} Jul 05 13:38:38.036254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.036182 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-182.ec2.internal" podStartSLOduration=4.036160351 podStartE2EDuration="4.036160351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:38.035749061 +0000 UTC m=+6.045218686" watchObservedRunningTime="2026-07-05 13:38:38.036160351 +0000 UTC m=+6.045629969" Jul 05 13:38:38.036471 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.036415 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-182.ec2.internal" podStartSLOduration=4.036407175 podStartE2EDuration="4.036407175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:36.889230484 +0000 UTC m=+4.898700111" watchObservedRunningTime="2026-07-05 13:38:38.036407175 +0000 UTC m=+6.045876800" Jul 05 13:38:38.302815 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.302068 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:38.302815 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.302316 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:38.302815 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.302389 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:42.302367719 +0000 UTC m=+10.311837344 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:38.403377 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.402707 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:38.403377 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.402940 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:38.403377 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.403008 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:42.40298684 +0000 UTC m=+10.412456444 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:38.504097 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.503322 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:38.504097 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.503524 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:38:38.504097 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.503549 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:38:38.504097 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.503564 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:38.504097 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.503636 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:38:42.503614491 +0000 UTC m=+10.513084093 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:38.806252 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.805558 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:38.806252 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.805699 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:38.806252 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:38.806099 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:38.806252 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:38.806196 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:39.793697 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:39.793646 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:39.793886 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:39.793787 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:40.794393 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:40.794347 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:40.794875 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:40.794539 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:40.795059 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:40.795034 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:40.795191 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:40.795169 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:41.793711 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:41.793671 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:41.793929 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:41.793822 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:42.336687 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:42.336627 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:42.337194 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.336788 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:42.337194 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.336863 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:50.336841223 +0000 UTC m=+18.346310829 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:42.437164 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:42.437033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:42.437338 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.437220 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:42.437338 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.437296 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:38:50.437274282 +0000 UTC m=+18.446743883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:42.538252 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:42.538209 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:42.538481 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.538430 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:38:42.538481 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.538464 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:38:42.538620 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.538495 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:42.538620 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.538565 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:38:50.538545407 +0000 UTC m=+18.548015025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:42.794394 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:42.794279 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:42.794569 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.794410 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:42.794569 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:42.794493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:42.794693 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:42.794619 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:43.792996 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:43.792951 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:43.793527 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:43.793098 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:44.793492 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:44.793066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:44.793492 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:44.793208 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:44.793492 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:44.793455 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:44.794090 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:44.793581 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:45.793012 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:45.792976 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:45.793183 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:45.793086 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:46.793172 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:46.793113 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:46.793172 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:46.793168 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:46.793740 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:46.793253 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:46.793740 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:46.793388 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:47.793741 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:47.793702 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:47.794216 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:47.793840 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:48.793292 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:48.793252 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:48.793471 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:48.793418 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:48.793471 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:48.793459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:48.793612 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:48.793590 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:49.793310 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:49.793272 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:49.793807 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:49.793408 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:50.405082 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:50.405041 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:50.405346 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.405178 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:50.405346 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.405246 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:06.405227452 +0000 UTC m=+34.414697052 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:38:50.505420 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:50.505387 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:50.505605 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.505505 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:50.505605 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.505565 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:06.505550488 +0000 UTC m=+34.515020101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:38:50.606354 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:50.606316 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:50.606557 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.606505 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:38:50.606557 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.606530 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:38:50.606557 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.606544 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:50.606709 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.606614 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:39:06.606592406 +0000 UTC m=+34.616062025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:38:50.796134 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:50.796056 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:50.796563 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.796174 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:50.796563 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:50.796230 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:50.796563 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:50.796325 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:51.793688 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:51.793598 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:51.793846 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:51.793722 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:52.796863 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:52.796827 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:52.796863 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:52.796853 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:52.797395 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:52.796969 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:52.797395 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:52.797104 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:53.793125 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:53.793076 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:53.793320 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:53.793231 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:54.792970 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:54.792927 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:54.793461 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:54.792933 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:54.793461 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:54.793076 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:54.793461 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:54.793131 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:55.794129 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:55.793855 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:55.794941 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:55.794249 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:56.062911 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.062875 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-99jnj" event={"ID":"a98320c7-b668-4c1f-bfd6-7afe069bb368","Type":"ContainerStarted","Data":"d51bf192154520ce9c083c124f2b8350ecb7c7bb9f570092d5fe8ef5e2bfae8b"} Jul 05 13:38:56.064240 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.064218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zdz5r" event={"ID":"cce826b0-2bf5-4efc-a2dc-ef404b5449cf","Type":"ContainerStarted","Data":"d67cf4585b820ade5a0b0ac445fa187b718c6761d411d744da1dcb703d26a231"} Jul 05 13:38:56.065717 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.065693 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jq5sl" event={"ID":"e7f25710-f553-4569-8644-826c6adb6727","Type":"ContainerStarted","Data":"c593eccf4c46f0fd3387e1128f993a3e8a80a0338228ad7c270705f49f4fcb77"} Jul 05 13:38:56.067119 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.067095 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vrzwb" event={"ID":"6f72a14a-0f23-403a-8410-3085f801802e","Type":"ContainerStarted","Data":"2168b34dc10ece0518e51de8af91817c959fcfbe063b5bda53a3eee02b92fcf5"} Jul 05 13:38:56.068537 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.068512 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8tp7h" event={"ID":"82a3c6df-c47e-4738-a4fd-01179772eb73","Type":"ContainerStarted","Data":"69d9954eb19d403aad2a589d1fb62dc2ef1d6ffc4d9771b9b1f88fee09d965de"} Jul 05 13:38:56.070775 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.070757 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"a5f5c3701081290df1b5d45effcc318767fe3f84330b54c68de40e5d9b94f249"} Jul 05 13:38:56.070867 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.070855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"cd80cb5cc72794e850a90286e5b688582c808f9733cf03142403dee246af1bfa"} Jul 05 13:38:56.070928 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.070908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"d9b04bf7b43133e353ae9ef10e78906a7f4915bdc9e9a7bed36dd50770784edb"} Jul 05 13:38:56.070978 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.070956 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"effbc3de5f23154f77557301a6647cce916253b747bda0a83e70d4dcd4f95704"} Jul 05 13:38:56.071046 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.071004 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"c66afa6f3eb515af2a715038616eac386fcd454c6f36c2e3ea800dbcad9d39ce"} Jul 05 13:38:56.072229 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.072207 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f07c2e-c429-43df-840a-b38476bd199c" containerID="0c6a23d1f53cc6aa2ad668819309c16996ac153139190b7b9533c71647cbbf89" exitCode=0 Jul 05 13:38:56.072309 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.072282 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerDied","Data":"0c6a23d1f53cc6aa2ad668819309c16996ac153139190b7b9533c71647cbbf89"} Jul 05 13:38:56.073869 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.073848 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" event={"ID":"a5df9764-35a8-4d25-9354-dccd243658cc","Type":"ContainerStarted","Data":"1c7980f0340451e9a001520e5b16e993df3cfe7675718420cfa41dadef3c7cf3"} Jul 05 13:38:56.080519 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.080467 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-99jnj" podStartSLOduration=2.654250912 podStartE2EDuration="22.080455249s" totalImagesPullingTime="19.426204337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:56.08021757 +0000 UTC m=+24.089687201" watchObservedRunningTime="2026-07-05 13:38:56.080455249 +0000 UTC m=+24.089924869" Jul 05 13:38:56.147762 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.146721 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zdz5r" podStartSLOduration=11.829455132 podStartE2EDuration="23.146700557s" totalImagesPullingTime="11.317245425s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:56.14667132 +0000 UTC m=+24.156140941" watchObservedRunningTime="2026-07-05 13:38:56.146700557 +0000 UTC m=+24.156170180" Jul 05 13:38:56.149438 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.148600 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8tp7h" podStartSLOduration=10.829328525 podStartE2EDuration="22.148564232s" totalImagesPullingTime="11.319235707s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:56.125944182 +0000 UTC m=+24.135413801" watchObservedRunningTime="2026-07-05 13:38:56.148564232 +0000 UTC m=+24.158033844" Jul 05 13:38:56.166943 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.166887 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jq5sl" podStartSLOduration=2.694051444 podStartE2EDuration="22.166873278s" totalImagesPullingTime="19.472821834s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:56.166234386 +0000 UTC m=+24.175704002" watchObservedRunningTime="2026-07-05 13:38:56.166873278 +0000 UTC m=+24.176342897" Jul 05 13:38:56.793804 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.793730 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:56.794058 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:56.793730 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:56.794058 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:56.793828 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:56.794058 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:56.794008 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:57.077023 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.076988 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-765z7" event={"ID":"5bbf35c9-8555-4999-8b67-38431ab8d634","Type":"ContainerStarted","Data":"fa7e61d0ad4a4981c29f842e3dc1afa4813f5853e91ba88f5035d371942eaa43"} Jul 05 13:38:57.080259 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.080197 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"63752f436a3c4e1783041fca077efe2e4ae71cf11dfa5ba8b288d5b075ed2268"} Jul 05 13:38:57.100741 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.100688 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-765z7" podStartSLOduration=11.776106356 podStartE2EDuration="23.100658021s" totalImagesPullingTime="11.324551665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:57.100590193 +0000 UTC m=+25.110059813" watchObservedRunningTime="2026-07-05 13:38:57.100658021 +0000 UTC m=+25.110127643" Jul 05 13:38:57.100969 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.100762 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vrzwb" podStartSLOduration=12.820882827 podStartE2EDuration="24.100756919s" totalImagesPullingTime="11.279874092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:56.185138936 +0000 UTC m=+24.194608555" watchObservedRunningTime="2026-07-05 13:38:57.100756919 +0000 UTC m=+25.110226601" Jul 05 13:38:57.121841 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.121652 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 05 13:38:57.728425 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.728286 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-05T13:38:57.121835402Z","UUID":"cfeb9ca1-8f91-482f-8641-7d50185dad29","Handler":null,"Name":"","Endpoint":""} Jul 05 13:38:57.730092 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.730071 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 05 13:38:57.730226 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.730112 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 05 13:38:57.793636 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:57.793601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:57.793821 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:57.793737 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:38:58.084499 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:58.084407 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" event={"ID":"a5df9764-35a8-4d25-9354-dccd243658cc","Type":"ContainerStarted","Data":"3aefaae90b1fad1003598d5d8340d4b5ef4ee651a88aa93d98cc3f557c695d00"} Jul 05 13:38:58.793335 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:58.793253 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:38:58.793572 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:58.793386 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:38:58.793572 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:58.793425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:38:58.793572 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:58.793520 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:38:59.091280 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:59.091182 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"7c054a1ec0b06c59689159e1a4fa48ac1f75a1fca59d15a237bbaa42c51f9b25"} Jul 05 13:38:59.093514 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:59.093482 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" event={"ID":"a5df9764-35a8-4d25-9354-dccd243658cc","Type":"ContainerStarted","Data":"65d0b75b28f349c439f61d476c63f5f899c4f666fded81a7f606d2d92cdcd8e2"} Jul 05 13:38:59.114100 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:59.114049 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wn4tb" podStartSLOduration=12.280404329 podStartE2EDuration="26.114034055s" totalImagesPullingTime="13.833629726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:38:59.113748782 +0000 UTC m=+27.123218402" watchObservedRunningTime="2026-07-05 13:38:59.114034055 +0000 UTC m=+27.123503675" Jul 05 13:38:59.191332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:59.191296 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:59.192136 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:59.192108 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:38:59.793346 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:38:59.793311 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:38:59.793595 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:38:59.793437 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:39:00.096077 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:00.095907 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:39:00.096721 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:00.096351 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vrzwb" Jul 05 13:39:00.794002 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:00.793913 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:00.794165 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:00.794053 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:39:00.794165 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:00.794111 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:00.794271 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:00.794230 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:39:01.101731 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:01.101474 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" event={"ID":"9694b03a-00a4-4b05-a482-fb25462145f1","Type":"ContainerStarted","Data":"4853ff23c678914a55a765dd3c6c79fda71c650428d26bd59e223bbdae0d79f0"} Jul 05 13:39:01.130785 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:01.130584 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" podStartSLOduration=8.633978942 podStartE2EDuration="28.130563796s" totalImagesPullingTime="19.496584854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:39:01.12974832 +0000 UTC m=+29.139217964" watchObservedRunningTime="2026-07-05 13:39:01.130563796 +0000 UTC m=+29.140033417" Jul 05 13:39:01.793528 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:01.793493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:01.793702 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:01.793595 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:39:02.105823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.105746 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f07c2e-c429-43df-840a-b38476bd199c" containerID="f88cf1374a6c04cd46448dfde4b983f2173517ca09b4770682333d170ab9ff3b" exitCode=0 Jul 05 13:39:02.105823 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.105809 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerDied","Data":"f88cf1374a6c04cd46448dfde4b983f2173517ca09b4770682333d170ab9ff3b"} Jul 05 13:39:02.106307 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.106259 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:39:02.106388 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.106315 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:39:02.106388 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.106361 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:39:02.122286 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.122260 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:39:02.122496 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.122480 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:39:02.794789 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.794756 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:02.795045 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:02.794862 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:39:02.795045 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.794918 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:02.795045 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:02.795033 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:39:02.910665 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.910628 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbfxn"] Jul 05 13:39:02.914169 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.914118 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgg8b"] Jul 05 13:39:02.914879 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.914852 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcct8"] Jul 05 13:39:02.915003 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:02.914972 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:02.915177 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:02.915073 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:39:03.107560 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:03.107489 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:03.107945 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:03.107494 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:03.107945 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:03.107596 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:39:03.107945 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:03.107670 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:39:04.111688 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:04.111495 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f07c2e-c429-43df-840a-b38476bd199c" containerID="cd4d0f7d7ac8aa0d529d8b20c1e166340ed95bd1706f700aa2d49998198690d2" exitCode=0 Jul 05 13:39:04.111688 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:04.111585 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerDied","Data":"cd4d0f7d7ac8aa0d529d8b20c1e166340ed95bd1706f700aa2d49998198690d2"} Jul 05 13:39:04.793169 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:04.793137 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:04.793343 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:04.793146 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:04.793343 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:04.793267 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:39:04.793343 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:04.793320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:04.793535 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:04.793436 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:39:04.793535 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:04.793506 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:39:06.118261 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.118232 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f07c2e-c429-43df-840a-b38476bd199c" containerID="6ce3fbf5e987d367e20811636d640684c1d5504aa6033bdcd836bc4003f41f71" exitCode=0 Jul 05 13:39:06.118843 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.118275 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerDied","Data":"6ce3fbf5e987d367e20811636d640684c1d5504aa6033bdcd836bc4003f41f71"} Jul 05 13:39:06.426746 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.426668 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:06.426885 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.426775 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:39:06.426885 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.426830 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:38.42681683 +0000 UTC m=+66.436286441 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 05 13:39:06.527751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.527721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:06.527915 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.527888 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 05 13:39:06.527957 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.527950 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret podName:f2cb94bd-1a62-46db-b988-8726f95cea99 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:38.527936133 +0000 UTC m=+66.537405731 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret") pod "global-pull-secret-syncer-hgg8b" (UID: "f2cb94bd-1a62-46db-b988-8726f95cea99") : object "kube-system"/"original-pull-secret" not registered Jul 05 13:39:06.628452 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.628411 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:06.628700 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.628595 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 05 13:39:06.628700 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.628618 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 05 13:39:06.628700 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.628628 2569 projected.go:196] Error preparing data for projected volume kube-api-access-dstld for pod openshift-network-diagnostics/network-check-target-zcct8: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:39:06.628700 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.628687 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld podName:06b51e8e-f7d5-4c5d-a3ca-51603de557ac nodeName:}" failed. No retries permitted until 2026-07-05 13:39:38.628670265 +0000 UTC m=+66.638139880 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dstld" (UniqueName: "kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld") pod "network-check-target-zcct8" (UID: "06b51e8e-f7d5-4c5d-a3ca-51603de557ac") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 05 13:39:06.793670 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.793633 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:06.793670 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.793660 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:06.793927 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.793782 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-hgg8b" podUID="f2cb94bd-1a62-46db-b988-8726f95cea99" Jul 05 13:39:06.793927 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.793868 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:39:06.793927 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:06.793914 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:06.794080 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:06.793999 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zcct8" podUID="06b51e8e-f7d5-4c5d-a3ca-51603de557ac" Jul 05 13:39:08.271516 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.271394 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-182.ec2.internal" event="NodeReady" Jul 05 13:39:08.272115 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.271533 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 05 13:39:08.308099 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.308059 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bd47dd89-tn97n"] Jul 05 13:39:08.311323 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.311295 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm"] Jul 05 13:39:08.311490 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.311464 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.314792 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.314767 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4"] Jul 05 13:39:08.314915 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.314897 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" Jul 05 13:39:08.315202 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.315177 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 05 13:39:08.315384 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.315310 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 05 13:39:08.315803 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.315649 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x86wj\"" Jul 05 13:39:08.316624 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.316602 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 05 13:39:08.318642 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.318624 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:08.321034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.320851 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 05 13:39:08.321034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.320903 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 05 13:39:08.321034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.320927 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-wdbdp\"" Jul 05 13:39:08.321849 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.321808 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 05 13:39:08.322224 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.322047 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 05 13:39:08.322509 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.322483 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j7rrk\"" Jul 05 13:39:08.323057 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.323036 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 05 13:39:08.326530 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.326458 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm"] Jul 05 13:39:08.327308 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.327286 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lm677"] Jul 05 13:39:08.331173 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.331111 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jlm49"] Jul 05 13:39:08.331620 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.331345 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.336749 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.336722 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 05 13:39:08.337403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.337358 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 05 13:39:08.337581 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.337565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sw56z\"" Jul 05 13:39:08.338189 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.338163 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4"] Jul 05 13:39:08.338189 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.338196 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bd47dd89-tn97n"] Jul 05 13:39:08.338403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.338289 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.342440 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.341944 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 05 13:39:08.342440 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.342165 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 05 13:39:08.342440 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.342401 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 05 13:39:08.342673 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.342444 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lm677"] Jul 05 13:39:08.342673 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.342641 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-mls4n\"" Jul 05 13:39:08.349542 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.349500 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jlm49"] Jul 05 13:39:08.440339 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-bound-sa-token\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440339 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3730e0ef-cd9d-401d-ab49-9261e28e14b5-cert\") pod \"ingress-canary-jlm49\" (UID: \"3730e0ef-cd9d-401d-ab49-9261e28e14b5\") " pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.440566 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxwb\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-kube-api-access-tkxwb\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440566 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440466 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-installation-pull-secrets\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440566 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm2cf\" (UniqueName: \"kubernetes.io/projected/3730e0ef-cd9d-401d-ab49-9261e28e14b5-kube-api-access-jm2cf\") pod \"ingress-canary-jlm49\" (UID: \"3730e0ef-cd9d-401d-ab49-9261e28e14b5\") " pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.440705 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-image-registry-private-configuration\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440705 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73359273-7ce0-4963-8590-b4e07880c348-ca-trust-extracted\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440705 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440634 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dblt\" (UniqueName: \"kubernetes.io/projected/c18c9f5e-4975-4885-8a88-4704fd516779-kube-api-access-7dblt\") pod \"network-check-source-989f8b8dc-79mnm\" (UID: \"c18c9f5e-4975-4885-8a88-4704fd516779\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" Jul 05 13:39:08.440705 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chgnh\" (UniqueName: \"kubernetes.io/projected/8e146763-8e15-4664-b91d-b527657be7bb-kube-api-access-chgnh\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.440705 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440916 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-trusted-ca\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440916 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440765 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-registry-certificates\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.440916 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e146763-8e15-4664-b91d-b527657be7bb-tmp-dir\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.440916 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e146763-8e15-4664-b91d-b527657be7bb-config-volume\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.440916 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440859 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e45a90da-f828-44b2-9627-eb68a5f8e4c8-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:08.440916 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440882 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.441139 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.440924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:08.541732 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:08.541732 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-bound-sa-token\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3730e0ef-cd9d-401d-ab49-9261e28e14b5-cert\") pod \"ingress-canary-jlm49\" (UID: \"3730e0ef-cd9d-401d-ab49-9261e28e14b5\") " pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541820 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxwb\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-kube-api-access-tkxwb\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-installation-pull-secrets\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm2cf\" (UniqueName: \"kubernetes.io/projected/3730e0ef-cd9d-401d-ab49-9261e28e14b5-kube-api-access-jm2cf\") pod \"ingress-canary-jlm49\" (UID: \"3730e0ef-cd9d-401d-ab49-9261e28e14b5\") " pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.541906 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-image-registry-private-configuration\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.541959 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.541959 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73359273-7ce0-4963-8590-b4e07880c348-ca-trust-extracted\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.541978 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:09.041956738 +0000 UTC m=+37.051426353 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dblt\" (UniqueName: \"kubernetes.io/projected/c18c9f5e-4975-4885-8a88-4704fd516779-kube-api-access-7dblt\") pod \"network-check-source-989f8b8dc-79mnm\" (UID: \"c18c9f5e-4975-4885-8a88-4704fd516779\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chgnh\" (UniqueName: \"kubernetes.io/projected/8e146763-8e15-4664-b91d-b527657be7bb-kube-api-access-chgnh\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542094 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-trusted-ca\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-registry-certificates\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542178 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e146763-8e15-4664-b91d-b527657be7bb-tmp-dir\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e146763-8e15-4664-b91d-b527657be7bb-config-volume\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e45a90da-f828-44b2-9627-eb68a5f8e4c8-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:08.542283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542272 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.542778 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.542327 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73359273-7ce0-4963-8590-b4e07880c348-ca-trust-extracted\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.542778 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.542405 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:08.542778 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.542454 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:39:09.042438821 +0000 UTC m=+37.051908423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:08.542778 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.542718 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:08.542778 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.542734 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:08.542778 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:08.542778 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:09.042762073 +0000 UTC m=+37.052231687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:08.543017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.543006 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e146763-8e15-4664-b91d-b527657be7bb-tmp-dir\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.543669 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.543151 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-registry-certificates\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.543669 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.543625 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8e146763-8e15-4664-b91d-b527657be7bb-config-volume\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.543805 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.543699 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e45a90da-f828-44b2-9627-eb68a5f8e4c8-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:08.543883 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.543864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-trusted-ca\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.546638 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.546616 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-installation-pull-secrets\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.546744 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.546695 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-image-registry-private-configuration\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.546913 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.546895 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3730e0ef-cd9d-401d-ab49-9261e28e14b5-cert\") pod \"ingress-canary-jlm49\" (UID: \"3730e0ef-cd9d-401d-ab49-9261e28e14b5\") " pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.551941 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.551808 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-bound-sa-token\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.551941 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.551887 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dblt\" (UniqueName: \"kubernetes.io/projected/c18c9f5e-4975-4885-8a88-4704fd516779-kube-api-access-7dblt\") pod \"network-check-source-989f8b8dc-79mnm\" (UID: \"c18c9f5e-4975-4885-8a88-4704fd516779\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" Jul 05 13:39:08.552586 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.552565 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxwb\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-kube-api-access-tkxwb\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:08.552711 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.552689 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm2cf\" (UniqueName: \"kubernetes.io/projected/3730e0ef-cd9d-401d-ab49-9261e28e14b5-kube-api-access-jm2cf\") pod \"ingress-canary-jlm49\" (UID: \"3730e0ef-cd9d-401d-ab49-9261e28e14b5\") " pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.554452 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.554414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chgnh\" (UniqueName: \"kubernetes.io/projected/8e146763-8e15-4664-b91d-b527657be7bb-kube-api-access-chgnh\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:08.638161 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.638121 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" Jul 05 13:39:08.660079 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.659793 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jlm49" Jul 05 13:39:08.793302 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.793196 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:08.793302 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.793196 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:08.793535 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.793198 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:08.796773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.796475 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 05 13:39:08.796773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.796557 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xhx9m\"" Jul 05 13:39:08.796773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.796618 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 05 13:39:08.796773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.796768 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fvggw\"" Jul 05 13:39:08.797268 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.797248 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm"] Jul 05 13:39:08.800838 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:39:08.800812 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18c9f5e_4975_4885_8a88_4704fd516779.slice/crio-e8e2caf47acae5663a41fb55c8c0f4f49d42da8dc0d4bd2beb399c885af8e4af WatchSource:0}: Error finding container e8e2caf47acae5663a41fb55c8c0f4f49d42da8dc0d4bd2beb399c885af8e4af: Status 404 returned error can't find the container with id e8e2caf47acae5663a41fb55c8c0f4f49d42da8dc0d4bd2beb399c885af8e4af Jul 05 13:39:08.808421 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:08.808398 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jlm49"] Jul 05 13:39:08.812433 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:39:08.812345 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3730e0ef_cd9d_401d_ab49_9261e28e14b5.slice/crio-c1ede28c4dd990f8d54632005095ae15dd0214fb80c199fd52c6523f7e044b0a WatchSource:0}: Error finding container c1ede28c4dd990f8d54632005095ae15dd0214fb80c199fd52c6523f7e044b0a: Status 404 returned error can't find the container with id c1ede28c4dd990f8d54632005095ae15dd0214fb80c199fd52c6523f7e044b0a Jul 05 13:39:09.048108 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:09.048024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:09.048108 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:09.048078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048169 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:09.048174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048232 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:39:10.04821453 +0000 UTC m=+38.057684136 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048248 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048263 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048275 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048320 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:10.048309039 +0000 UTC m=+38.057778651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:09.048331 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:09.048336 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:10.048326891 +0000 UTC m=+38.057796494 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:09.127939 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:09.127902 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jlm49" event={"ID":"3730e0ef-cd9d-401d-ab49-9261e28e14b5","Type":"ContainerStarted","Data":"c1ede28c4dd990f8d54632005095ae15dd0214fb80c199fd52c6523f7e044b0a"} Jul 05 13:39:09.129198 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:09.129173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" event={"ID":"c18c9f5e-4975-4885-8a88-4704fd516779","Type":"ContainerStarted","Data":"e8e2caf47acae5663a41fb55c8c0f4f49d42da8dc0d4bd2beb399c885af8e4af"} Jul 05 13:39:10.059601 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:10.059562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:10.059663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:10.059744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.059764 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.059848 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:12.059830741 +0000 UTC m=+40.069300343 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.059858 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.059929 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:39:12.059910446 +0000 UTC m=+40.069380051 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.059952 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.059970 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:10.060087 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:10.060020 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:12.060005216 +0000 UTC m=+40.069474839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:11.138923 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:11.138857 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f07c2e-c429-43df-840a-b38476bd199c" containerID="fde88c59a7ec8497cad2992886f027b6ae4b3bc77e0d21eb18e6f6288614bfe0" exitCode=0 Jul 05 13:39:11.138923 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:11.138908 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerDied","Data":"fde88c59a7ec8497cad2992886f027b6ae4b3bc77e0d21eb18e6f6288614bfe0"} Jul 05 13:39:12.078342 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:12.078132 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:12.078342 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078329 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:12.078342 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078353 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:12.078640 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:12.078402 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:12.078640 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078443 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:16.078422692 +0000 UTC m=+44.087892313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:12.078640 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:12.078490 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:12.078640 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078513 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:12.078640 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078558 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:39:16.078544552 +0000 UTC m=+44.088014152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:12.078640 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078601 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:12.078837 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:12.078652 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:16.078638235 +0000 UTC m=+44.088107847 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:12.144825 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:12.144785 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8f07c2e-c429-43df-840a-b38476bd199c" containerID="7eb6b1980070b581576e83a1fc22a358797a4e0ae3f53d00427f66a16f194fa4" exitCode=0 Jul 05 13:39:12.145292 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:12.144843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerDied","Data":"7eb6b1980070b581576e83a1fc22a358797a4e0ae3f53d00427f66a16f194fa4"} Jul 05 13:39:14.151443 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:14.151398 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jlm49" event={"ID":"3730e0ef-cd9d-401d-ab49-9261e28e14b5","Type":"ContainerStarted","Data":"7b8f759762e19bb8bcb93361ff417b2e4080c3211971c526003ea7d8012faecd"} Jul 05 13:39:14.153420 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:14.153393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" event={"ID":"c18c9f5e-4975-4885-8a88-4704fd516779","Type":"ContainerStarted","Data":"1955e8808448f033d555a4334cc74dfe7c028bc5218e6a67b0d3df582f3041ab"} Jul 05 13:39:14.156985 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:14.156961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7229l" event={"ID":"b8f07c2e-c429-43df-840a-b38476bd199c","Type":"ContainerStarted","Data":"f2874ec24cfa99e22981f671fdbe89616f0474eb66a21cc9662e854d70cb6511"} Jul 05 13:39:14.169505 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:14.169398 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jlm49" podStartSLOduration=1.040258904 podStartE2EDuration="6.169361782s" totalImagesPullingTime="5.129102878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:39:14.16853148 +0000 UTC m=+42.178001120" watchObservedRunningTime="2026-07-05 13:39:14.169361782 +0000 UTC m=+42.178831401" Jul 05 13:39:14.185791 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:14.185737 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-79mnm" podStartSLOduration=35.032761924 podStartE2EDuration="40.185720837s" totalImagesPullingTime="5.152958913s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:39:14.18508782 +0000 UTC m=+42.194557462" watchObservedRunningTime="2026-07-05 13:39:14.185720837 +0000 UTC m=+42.195190457" Jul 05 13:39:14.207259 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:14.207209 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7229l" podStartSLOduration=17.186126044 podStartE2EDuration="40.207193584s" totalImagesPullingTime="23.02106754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:39:14.206072052 +0000 UTC m=+42.215541711" watchObservedRunningTime="2026-07-05 13:39:14.207193584 +0000 UTC m=+42.216663203" Jul 05 13:39:16.109358 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:16.109319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:16.109382 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:16.109418 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109479 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109501 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109539 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109558 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:24.109541248 +0000 UTC m=+52.119010863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109585 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:24.109573788 +0000 UTC m=+52.119043387 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109539 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:16.109734 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:16.109617 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:39:24.109607602 +0000 UTC m=+52.119077202 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:24.170639 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:24.170606 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:24.170659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:24.170744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170762 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170811 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170824 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:39:40.170805259 +0000 UTC m=+68.180274860 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170852 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170859 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:40.17084567 +0000 UTC m=+68.180315296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170865 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:24.171173 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:24.170912 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:39:40.170899456 +0000 UTC m=+68.180369072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:34.130531 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:34.130502 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cgzlg" Jul 05 13:39:38.480037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.479993 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:39:38.483199 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.483179 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 05 13:39:38.490644 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:38.490620 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 05 13:39:38.490753 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:38.490694 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:40:42.490672343 +0000 UTC m=+130.500141955 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : secret "metrics-daemon-secret" not found Jul 05 13:39:38.581412 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.581354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:38.584587 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.584567 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 05 13:39:38.595753 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.595727 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f2cb94bd-1a62-46db-b988-8726f95cea99-original-pull-secret\") pod \"global-pull-secret-syncer-hgg8b\" (UID: \"f2cb94bd-1a62-46db-b988-8726f95cea99\") " pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:38.682723 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.682686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:38.685141 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.685119 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dstld\" (UniqueName: \"kubernetes.io/projected/06b51e8e-f7d5-4c5d-a3ca-51603de557ac-kube-api-access-dstld\") pod \"network-check-target-zcct8\" (UID: \"06b51e8e-f7d5-4c5d-a3ca-51603de557ac\") " pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:38.809616 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.809537 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xhx9m\"" Jul 05 13:39:38.815937 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.815917 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-hgg8b" Jul 05 13:39:38.816995 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.816977 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:38.947012 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.946979 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-hgg8b"] Jul 05 13:39:38.950173 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:39:38.950145 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2cb94bd_1a62_46db_b988_8726f95cea99.slice/crio-16f197f107494926ecac871f63bf5929d5febaa34fa798f9aca6956194c6f1b8 WatchSource:0}: Error finding container 16f197f107494926ecac871f63bf5929d5febaa34fa798f9aca6956194c6f1b8: Status 404 returned error can't find the container with id 16f197f107494926ecac871f63bf5929d5febaa34fa798f9aca6956194c6f1b8 Jul 05 13:39:38.961605 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:38.961581 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zcct8"] Jul 05 13:39:38.963847 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:39:38.963825 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06b51e8e_f7d5_4c5d_a3ca_51603de557ac.slice/crio-fe2aabbb9be75256bdd527e4eb39f15dfd41821ffdea16f79418fb5a8198e789 WatchSource:0}: Error finding container fe2aabbb9be75256bdd527e4eb39f15dfd41821ffdea16f79418fb5a8198e789: Status 404 returned error can't find the container with id fe2aabbb9be75256bdd527e4eb39f15dfd41821ffdea16f79418fb5a8198e789 Jul 05 13:39:39.218076 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:39.218039 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcct8" event={"ID":"06b51e8e-f7d5-4c5d-a3ca-51603de557ac","Type":"ContainerStarted","Data":"4af8066d7873f25aa5ec57ea3336ae4752832fa2b7c04799dfb4cb2bc3210118"} Jul 05 13:39:39.218259 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:39.218090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zcct8" event={"ID":"06b51e8e-f7d5-4c5d-a3ca-51603de557ac","Type":"ContainerStarted","Data":"fe2aabbb9be75256bdd527e4eb39f15dfd41821ffdea16f79418fb5a8198e789"} Jul 05 13:39:39.218259 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:39.218119 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:39:39.219187 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:39.219167 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgg8b" event={"ID":"f2cb94bd-1a62-46db-b988-8726f95cea99","Type":"ContainerStarted","Data":"16f197f107494926ecac871f63bf5929d5febaa34fa798f9aca6956194c6f1b8"} Jul 05 13:39:39.234647 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:39.234604 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zcct8" podStartSLOduration=65.234588168 podStartE2EDuration="1m5.234588168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:39:39.233853672 +0000 UTC m=+67.243323294" watchObservedRunningTime="2026-07-05 13:39:39.234588168 +0000 UTC m=+67.244057782" Jul 05 13:39:40.194687 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:40.194644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:39:40.194687 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:40.194694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194799 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194808 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194832 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:40.194842 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194856 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:40:12.194842483 +0000 UTC m=+100.204312095 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194897 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:40:12.194877136 +0000 UTC m=+100.204346739 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194925 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:39:40.195199 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:39:40.194967 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:40:12.194952893 +0000 UTC m=+100.204422492 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:39:44.235325 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:44.235289 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-hgg8b" event={"ID":"f2cb94bd-1a62-46db-b988-8726f95cea99","Type":"ContainerStarted","Data":"d802fd2ccddfff1d5ad3b3d3eaa6d82e3b3fbe583ff6b24cb729d28704c21fa5"} Jul 05 13:39:44.256532 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:39:44.256483 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-hgg8b" podStartSLOduration=65.948332916 podStartE2EDuration="1m10.256467296s" totalImagesPullingTime="4.30813438s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:39:44.255479399 +0000 UTC m=+72.264949053" watchObservedRunningTime="2026-07-05 13:39:44.256467296 +0000 UTC m=+72.265936916" Jul 05 13:40:10.224400 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:40:10.224351 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zcct8" Jul 05 13:40:12.229504 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:40:12.229463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:40:12.229517 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229602 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229611 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229632 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:40:12.229633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229690 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:41:16.229674877 +0000 UTC m=+164.239144475 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229704 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:41:16.229697585 +0000 UTC m=+164.239167184 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229767 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:40:12.229884 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:12.229829 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:41:16.229812643 +0000 UTC m=+164.239282258 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:40:42.552481 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:40:42.552441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:40:42.552987 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:42.552577 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 05 13:40:42.552987 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:40:42.552629 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:44.552615484 +0000 UTC m=+252.562085082 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : secret "metrics-daemon-secret" not found Jul 05 13:41:11.325942 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:11.325898 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" podUID="73359273-7ce0-4963-8590-b4e07880c348" Jul 05 13:41:11.344070 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:11.344033 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" podUID="e45a90da-f828-44b2-9627-eb68a5f8e4c8" Jul 05 13:41:11.351191 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:11.351167 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lm677" podUID="8e146763-8e15-4664-b91d-b527657be7bb" Jul 05 13:41:11.448327 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:11.448300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:41:11.448522 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:11.448299 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lm677" Jul 05 13:41:11.448522 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:11.448300 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:41:11.821944 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:11.821890 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:41:16.302867 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:16.302827 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:16.302875 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:16.302902 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.302995 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.302995 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.303062 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert podName:e45a90da-f828-44b2-9627-eb68a5f8e4c8 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:18.303047861 +0000 UTC m=+286.312517463 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-g8fr4" (UID: "e45a90da-f828-44b2-9627-eb68a5f8e4c8") : secret "networking-console-plugin-cert" not found Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.303107 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls podName:8e146763-8e15-4664-b91d-b527657be7bb nodeName:}" failed. No retries permitted until 2026-07-05 13:43:18.303092486 +0000 UTC m=+286.312562098 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls") pod "dns-default-lm677" (UID: "8e146763-8e15-4664-b91d-b527657be7bb") : secret "dns-default-metrics-tls" not found Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.302995 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.303127 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5bd47dd89-tn97n: secret "image-registry-tls" not found Jul 05 13:41:16.303337 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:41:16.303150 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls podName:73359273-7ce0-4963-8590-b4e07880c348 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:18.303144409 +0000 UTC m=+286.312614032 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls") pod "image-registry-5bd47dd89-tn97n" (UID: "73359273-7ce0-4963-8590-b4e07880c348") : secret "image-registry-tls" not found Jul 05 13:41:22.794765 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:41:22.794724 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:42:44.599295 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:44.599246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:42:44.599779 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:44.599385 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 05 13:42:44.599779 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:44.599443 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs podName:1d342df4-57dc-415e-9232-a44b96de4d37 nodeName:}" failed. No retries permitted until 2026-07-05 13:44:46.599428889 +0000 UTC m=+374.608898486 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs") pod "network-metrics-daemon-jbfxn" (UID: "1d342df4-57dc-415e-9232-a44b96de4d37") : secret "metrics-daemon-secret" not found Jul 05 13:42:55.589847 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.589769 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6"] Jul 05 13:42:55.592562 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.592548 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.596227 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.596194 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xrkbr"] Jul 05 13:42:55.596566 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.596549 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-pcl6n\"" Jul 05 13:42:55.598570 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.598551 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 05 13:42:55.598706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.598565 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 05 13:42:55.598706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.598601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 05 13:42:55.598706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.598621 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 05 13:42:55.598865 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.598771 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh"] Jul 05 13:42:55.598917 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.598866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.601595 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.601579 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 05 13:42:55.601797 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.601782 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:55.601893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.601878 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 05 13:42:55.602560 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.602546 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mb4h4\"" Jul 05 13:42:55.603648 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.603627 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 05 13:42:55.605137 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.605120 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 05 13:42:55.606332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.606171 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 05 13:42:55.606332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.606230 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 05 13:42:55.606332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.606229 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-6wjlk\"" Jul 05 13:42:55.606332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.606281 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6"] Jul 05 13:42:55.606597 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.606479 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 05 13:42:55.608619 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.608601 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 05 13:42:55.615892 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.615874 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh"] Jul 05 13:42:55.616593 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.616574 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xrkbr"] Jul 05 13:42:55.669062 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.669167 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669094 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrhw\" (UniqueName: \"kubernetes.io/projected/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-kube-api-access-grrhw\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.669236 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054b55e9-2ef5-4320-ad4c-c25d2c221495-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.669236 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669195 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz72n\" (UniqueName: \"kubernetes.io/projected/054b55e9-2ef5-4320-ad4c-c25d2c221495-kube-api-access-hz72n\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.669236 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/054b55e9-2ef5-4320-ad4c-c25d2c221495-serving-cert\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.669354 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669254 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8frz\" (UniqueName: \"kubernetes.io/projected/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-kube-api-access-j8frz\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:55.669354 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.669354 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054b55e9-2ef5-4320-ad4c-c25d2c221495-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.669495 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/054b55e9-2ef5-4320-ad4c-c25d2c221495-tmp\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.669495 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669428 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:55.669495 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.669450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/054b55e9-2ef5-4320-ad4c-c25d2c221495-snapshots\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.696026 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.696005 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb"] Jul 05 13:42:55.698680 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.698663 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8446587799-l6kwv"] Jul 05 13:42:55.698793 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.698777 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.701342 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.701325 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.702907 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.702878 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 05 13:42:55.702999 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.702951 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 05 13:42:55.703080 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.703058 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 05 13:42:55.703756 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.703725 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-cqj7l\"" Jul 05 13:42:55.704034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.703991 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 05 13:42:55.704735 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.704717 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 05 13:42:55.706086 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.706021 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 05 13:42:55.706281 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.706262 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 05 13:42:55.706934 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.706913 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 05 13:42:55.707093 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.707073 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 05 13:42:55.707167 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.707130 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-tpcp6\"" Jul 05 13:42:55.707826 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.707805 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 05 13:42:55.710027 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.710005 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb"] Jul 05 13:42:55.711778 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.711757 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8446587799-l6kwv"] Jul 05 13:42:55.770056 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.770164 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrhw\" (UniqueName: \"kubernetes.io/projected/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-kube-api-access-grrhw\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.770164 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770132 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-stats-auth\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.770164 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770161 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054b55e9-2ef5-4320-ad4c-c25d2c221495-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770325 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz72n\" (UniqueName: \"kubernetes.io/projected/054b55e9-2ef5-4320-ad4c-c25d2c221495-kube-api-access-hz72n\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770325 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770231 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-default-certificate\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.770325 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770252 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/054b55e9-2ef5-4320-ad4c-c25d2c221495-serving-cert\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770325 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770290 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8frz\" (UniqueName: \"kubernetes.io/projected/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-kube-api-access-j8frz\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:55.770541 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.770541 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2lt6\" (UniqueName: \"kubernetes.io/projected/19560aa3-2a88-468c-aa1c-0320e499a8ff-kube-api-access-h2lt6\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.770541 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054b55e9-2ef5-4320-ad4c-c25d2c221495-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770541 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770439 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/054b55e9-2ef5-4320-ad4c-c25d2c221495-tmp\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770541 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.770467 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:55.770541 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770494 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.770560 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls podName:a53248a8-e057-4cc5-8535-10e1ecbd3a7e nodeName:}" failed. No retries permitted until 2026-07-05 13:42:56.270539204 +0000 UTC m=+264.280008812 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-n6vq6" (UID: "a53248a8-e057-4cc5-8535-10e1ecbd3a7e") : secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770590 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770631 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/054b55e9-2ef5-4320-ad4c-c25d2c221495-snapshots\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.770711 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770733 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054b55e9-2ef5-4320-ad4c-c25d2c221495-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770713 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/054b55e9-2ef5-4320-ad4c-c25d2c221495-tmp\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.770781 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls podName:2bd348fc-b188-42e1-8a0d-06d9e4c4c033 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:56.270744706 +0000 UTC m=+264.280214318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-5q7zh" (UID: "2bd348fc-b188-42e1-8a0d-06d9e4c4c033") : secret "samples-operator-tls" not found Jul 05 13:42:55.770809 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770800 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2csn\" (UniqueName: \"kubernetes.io/projected/6cdd610b-7966-4c3a-9648-21dbcf1828b2-kube-api-access-q2csn\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.771195 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.771195 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770866 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19560aa3-2a88-468c-aa1c-0320e499a8ff-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.771195 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.770891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19560aa3-2a88-468c-aa1c-0320e499a8ff-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.771352 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.771238 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/054b55e9-2ef5-4320-ad4c-c25d2c221495-snapshots\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.771352 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.771278 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/054b55e9-2ef5-4320-ad4c-c25d2c221495-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.771613 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.771592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.772499 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.772483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/054b55e9-2ef5-4320-ad4c-c25d2c221495-serving-cert\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.779490 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.779469 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz72n\" (UniqueName: \"kubernetes.io/projected/054b55e9-2ef5-4320-ad4c-c25d2c221495-kube-api-access-hz72n\") pod \"insights-operator-5c847dcfc4-xrkbr\" (UID: \"054b55e9-2ef5-4320-ad4c-c25d2c221495\") " pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:55.779587 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.779509 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8frz\" (UniqueName: \"kubernetes.io/projected/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-kube-api-access-j8frz\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:55.779879 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.779860 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrhw\" (UniqueName: \"kubernetes.io/projected/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-kube-api-access-grrhw\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:55.871497 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-default-certificate\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.871497 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871483 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2lt6\" (UniqueName: \"kubernetes.io/projected/19560aa3-2a88-468c-aa1c-0320e499a8ff-kube-api-access-h2lt6\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.871621 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871511 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.871621 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2csn\" (UniqueName: \"kubernetes.io/projected/6cdd610b-7966-4c3a-9648-21dbcf1828b2-kube-api-access-q2csn\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.871621 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19560aa3-2a88-468c-aa1c-0320e499a8ff-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.871777 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871654 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19560aa3-2a88-468c-aa1c-0320e499a8ff-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.871777 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.871688 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 13:42:55.871777 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.871755 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:56.371735948 +0000 UTC m=+264.381205555 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : secret "router-metrics-certs-default" not found Jul 05 13:42:55.871777 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.871978 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.871826 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-stats-auth\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.871978 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:55.871957 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:56.371939801 +0000 UTC m=+264.381409424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : configmap references non-existent config key: service-ca.crt Jul 05 13:42:55.872234 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.872215 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/19560aa3-2a88-468c-aa1c-0320e499a8ff-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.873633 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.873610 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/19560aa3-2a88-468c-aa1c-0320e499a8ff-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.873790 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.873769 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-default-certificate\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.874066 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.874048 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-stats-auth\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.880670 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.880646 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2lt6\" (UniqueName: \"kubernetes.io/projected/19560aa3-2a88-468c-aa1c-0320e499a8ff-kube-api-access-h2lt6\") pod \"kube-storage-version-migrator-operator-846bf668f7-4d8vb\" (UID: \"19560aa3-2a88-468c-aa1c-0320e499a8ff\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:55.880839 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.880823 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2csn\" (UniqueName: \"kubernetes.io/projected/6cdd610b-7966-4c3a-9648-21dbcf1828b2-kube-api-access-q2csn\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:55.910888 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:55.910866 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" Jul 05 13:42:56.012682 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.012658 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" Jul 05 13:42:56.028675 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.028646 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-xrkbr"] Jul 05 13:42:56.031258 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:42:56.031231 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod054b55e9_2ef5_4320_ad4c_c25d2c221495.slice/crio-ed92bcdd9f1e712c30feb47c37834eaa8900544a86c3c4419f29524aa3e4ac66 WatchSource:0}: Error finding container ed92bcdd9f1e712c30feb47c37834eaa8900544a86c3c4419f29524aa3e4ac66: Status 404 returned error can't find the container with id ed92bcdd9f1e712c30feb47c37834eaa8900544a86c3c4419f29524aa3e4ac66 Jul 05 13:42:56.128518 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.128447 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb"] Jul 05 13:42:56.131161 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:42:56.131131 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19560aa3_2a88_468c_aa1c_0320e499a8ff.slice/crio-88a4c8160ff1252eecc241c4e8783213354875e25eb4e4e8578f9e46a3dd64bb WatchSource:0}: Error finding container 88a4c8160ff1252eecc241c4e8783213354875e25eb4e4e8578f9e46a3dd64bb: Status 404 returned error can't find the container with id 88a4c8160ff1252eecc241c4e8783213354875e25eb4e4e8578f9e46a3dd64bb Jul 05 13:42:56.274761 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.274735 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:56.274892 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.274860 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:56.274955 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.274886 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 13:42:56.274955 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.274937 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls podName:2bd348fc-b188-42e1-8a0d-06d9e4c4c033 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:57.274922177 +0000 UTC m=+265.284391775 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-5q7zh" (UID: "2bd348fc-b188-42e1-8a0d-06d9e4c4c033") : secret "samples-operator-tls" not found Jul 05 13:42:56.275052 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.274959 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:56.275052 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.275011 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls podName:a53248a8-e057-4cc5-8535-10e1ecbd3a7e nodeName:}" failed. No retries permitted until 2026-07-05 13:42:57.274995683 +0000 UTC m=+265.284465295 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-n6vq6" (UID: "a53248a8-e057-4cc5-8535-10e1ecbd3a7e") : secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:56.375586 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.375563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:56.375679 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.375625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:56.375723 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.375707 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:57.375692754 +0000 UTC m=+265.385162352 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : configmap references non-existent config key: service-ca.crt Jul 05 13:42:56.375789 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.375723 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 13:42:56.375789 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:56.375765 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:57.375755008 +0000 UTC m=+265.385224605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : secret "router-metrics-certs-default" not found Jul 05 13:42:56.697978 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.697945 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" event={"ID":"19560aa3-2a88-468c-aa1c-0320e499a8ff","Type":"ContainerStarted","Data":"88a4c8160ff1252eecc241c4e8783213354875e25eb4e4e8578f9e46a3dd64bb"} Jul 05 13:42:56.698877 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:56.698859 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" event={"ID":"054b55e9-2ef5-4320-ad4c-c25d2c221495","Type":"ContainerStarted","Data":"ed92bcdd9f1e712c30feb47c37834eaa8900544a86c3c4419f29524aa3e4ac66"} Jul 05 13:42:57.283239 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:57.283192 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:57.283445 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:57.283329 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:57.283445 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.283375 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 13:42:57.283572 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.283464 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:57.283572 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.283486 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls podName:2bd348fc-b188-42e1-8a0d-06d9e4c4c033 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:59.283463829 +0000 UTC m=+267.292933439 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-5q7zh" (UID: "2bd348fc-b188-42e1-8a0d-06d9e4c4c033") : secret "samples-operator-tls" not found Jul 05 13:42:57.283572 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.283514 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls podName:a53248a8-e057-4cc5-8535-10e1ecbd3a7e nodeName:}" failed. No retries permitted until 2026-07-05 13:42:59.283498577 +0000 UTC m=+267.292968179 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-n6vq6" (UID: "a53248a8-e057-4cc5-8535-10e1ecbd3a7e") : secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:57.384710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:57.384677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:57.384845 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:57.384750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:57.384895 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.384861 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:59.384842817 +0000 UTC m=+267.394312414 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : configmap references non-existent config key: service-ca.crt Jul 05 13:42:57.384937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.384902 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 13:42:57.384969 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:57.384943 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:42:59.384931882 +0000 UTC m=+267.394401480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : secret "router-metrics-certs-default" not found Jul 05 13:42:58.705536 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:58.705495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" event={"ID":"19560aa3-2a88-468c-aa1c-0320e499a8ff","Type":"ContainerStarted","Data":"8c89a42f8a3855063ef7834d2c88ff9f3ba68bfe3fba3b717444326bc5c632d0"} Jul 05 13:42:58.706922 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:58.706899 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" event={"ID":"054b55e9-2ef5-4320-ad4c-c25d2c221495","Type":"ContainerStarted","Data":"cdc5c25a68dfa7f1f3cad4947a1ade7897adef10e671fb2100eb6f2bbcbfd217"} Jul 05 13:42:58.725043 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:58.724958 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" podStartSLOduration=1.393712822 podStartE2EDuration="3.724940206s" totalImagesPullingTime="2.331227384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:42:58.724056139 +0000 UTC m=+266.733525758" watchObservedRunningTime="2026-07-05 13:42:58.724940206 +0000 UTC m=+266.734409829" Jul 05 13:42:58.744034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:58.743980 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" podStartSLOduration=1.315712025 podStartE2EDuration="3.743961424s" totalImagesPullingTime="2.428249399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:42:58.743909863 +0000 UTC m=+266.753379503" watchObservedRunningTime="2026-07-05 13:42:58.743961424 +0000 UTC m=+266.753431045" Jul 05 13:42:59.302972 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.302933 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:42:59.303141 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.303118 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:59.303222 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.303122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:42:59.303222 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.303195 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls podName:a53248a8-e057-4cc5-8535-10e1ecbd3a7e nodeName:}" failed. No retries permitted until 2026-07-05 13:43:03.303175375 +0000 UTC m=+271.312644979 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-n6vq6" (UID: "a53248a8-e057-4cc5-8535-10e1ecbd3a7e") : secret "cluster-monitoring-operator-tls" not found Jul 05 13:42:59.303222 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.303202 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 13:42:59.303359 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.303259 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls podName:2bd348fc-b188-42e1-8a0d-06d9e4c4c033 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:03.303242658 +0000 UTC m=+271.312712256 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-5q7zh" (UID: "2bd348fc-b188-42e1-8a0d-06d9e4c4c033") : secret "samples-operator-tls" not found Jul 05 13:42:59.403892 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.403863 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:59.404046 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.403929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:42:59.404046 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.404007 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 13:42:59.404046 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.404039 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:03.404019393 +0000 UTC m=+271.413488996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : configmap references non-existent config key: service-ca.crt Jul 05 13:42:59.404198 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.404069 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:03.404055487 +0000 UTC m=+271.413525089 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : secret "router-metrics-certs-default" not found Jul 05 13:42:59.643954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.641903 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qv25g"] Jul 05 13:42:59.646275 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.646252 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.649721 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.649697 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 05 13:42:59.649987 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.649969 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 05 13:42:59.651089 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.651072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jq2lc\"" Jul 05 13:42:59.658687 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.658663 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qv25g"] Jul 05 13:42:59.807848 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.807821 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.807848 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.807850 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnxwx\" (UniqueName: \"kubernetes.io/projected/78c320ec-5b42-4290-b828-cf373261e157-kube-api-access-tnxwx\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.808262 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.807977 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.808262 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.808133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78c320ec-5b42-4290-b828-cf373261e157-data-volume\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.808262 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.808250 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78c320ec-5b42-4290-b828-cf373261e157-crio-socket\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.908994 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.908925 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.908994 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.908955 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnxwx\" (UniqueName: \"kubernetes.io/projected/78c320ec-5b42-4290-b828-cf373261e157-kube-api-access-tnxwx\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.909182 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.909041 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 13:42:59.909182 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.909096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.909182 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.909119 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:00.409099085 +0000 UTC m=+268.418568691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : configmap "kube-rbac-proxy" not found Jul 05 13:42:59.909182 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.909180 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 13:42:59.909411 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.909206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78c320ec-5b42-4290-b828-cf373261e157-data-volume\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.909411 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:42:59.909242 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:00.409225353 +0000 UTC m=+268.418694967 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : secret "insights-runtime-extractor-tls" not found Jul 05 13:42:59.909411 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.909324 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78c320ec-5b42-4290-b828-cf373261e157-crio-socket\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.909521 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.909426 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/78c320ec-5b42-4290-b828-cf373261e157-crio-socket\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.909555 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.909538 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/78c320ec-5b42-4290-b828-cf373261e157-data-volume\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:42:59.917851 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:42:59.917832 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnxwx\" (UniqueName: \"kubernetes.io/projected/78c320ec-5b42-4290-b828-cf373261e157-kube-api-access-tnxwx\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:00.413210 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:00.413165 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:00.413389 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:00.413236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:00.413389 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:00.413284 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 13:43:00.413389 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:00.413331 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 13:43:00.413389 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:00.413354 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:01.413339159 +0000 UTC m=+269.422808761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : configmap "kube-rbac-proxy" not found Jul 05 13:43:00.413389 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:00.413389 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:01.413362517 +0000 UTC m=+269.422832118 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : secret "insights-runtime-extractor-tls" not found Jul 05 13:43:01.178291 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.178260 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89"] Jul 05 13:43:01.181270 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.181256 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" Jul 05 13:43:01.185288 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.185258 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 05 13:43:01.185288 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.185265 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 05 13:43:01.185496 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.185356 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-bvhmc\"" Jul 05 13:43:01.196908 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.196883 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89"] Jul 05 13:43:01.321818 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.321780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s7mk\" (UniqueName: \"kubernetes.io/projected/70591d88-9268-44bc-bcba-453202d409ba-kube-api-access-8s7mk\") pod \"migrator-d57d58cc9-wmd89\" (UID: \"70591d88-9268-44bc-bcba-453202d409ba\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" Jul 05 13:43:01.423187 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.423152 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:01.423327 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.423202 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s7mk\" (UniqueName: \"kubernetes.io/projected/70591d88-9268-44bc-bcba-453202d409ba-kube-api-access-8s7mk\") pod \"migrator-d57d58cc9-wmd89\" (UID: \"70591d88-9268-44bc-bcba-453202d409ba\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" Jul 05 13:43:01.423327 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.423245 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:01.423327 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:01.423289 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 13:43:01.423503 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:01.423344 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 13:43:01.423503 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:01.423357 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:03.423337329 +0000 UTC m=+271.432806934 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : configmap "kube-rbac-proxy" not found Jul 05 13:43:01.423503 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:01.423403 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:03.423392268 +0000 UTC m=+271.432861866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : secret "insights-runtime-extractor-tls" not found Jul 05 13:43:01.432414 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.432344 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s7mk\" (UniqueName: \"kubernetes.io/projected/70591d88-9268-44bc-bcba-453202d409ba-kube-api-access-8s7mk\") pod \"migrator-d57d58cc9-wmd89\" (UID: \"70591d88-9268-44bc-bcba-453202d409ba\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" Jul 05 13:43:01.489944 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.489913 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" Jul 05 13:43:01.521572 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.521546 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8tp7h_82a3c6df-c47e-4738-a4fd-01179772eb73/dns-node-resolver/0.log" Jul 05 13:43:01.611091 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.611001 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89"] Jul 05 13:43:01.613565 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:01.613536 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70591d88_9268_44bc_bcba_453202d409ba.slice/crio-fb45d2c43ed83c7be5722fab07a7cc1c9718272ae191d7f8771969e41ef498ca WatchSource:0}: Error finding container fb45d2c43ed83c7be5722fab07a7cc1c9718272ae191d7f8771969e41ef498ca: Status 404 returned error can't find the container with id fb45d2c43ed83c7be5722fab07a7cc1c9718272ae191d7f8771969e41ef498ca Jul 05 13:43:01.716552 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:01.716515 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" event={"ID":"70591d88-9268-44bc-bcba-453202d409ba","Type":"ContainerStarted","Data":"fb45d2c43ed83c7be5722fab07a7cc1c9718272ae191d7f8771969e41ef498ca"} Jul 05 13:43:02.920706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:02.920685 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zdz5r_cce826b0-2bf5-4efc-a2dc-ef404b5449cf/node-ca/0.log" Jul 05 13:43:03.337665 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.337628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:43:03.337837 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.337706 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:43:03.337837 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.337774 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 13:43:03.337936 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.337840 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls podName:a53248a8-e057-4cc5-8535-10e1ecbd3a7e nodeName:}" failed. No retries permitted until 2026-07-05 13:43:11.33782603 +0000 UTC m=+279.347295632 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-n6vq6" (UID: "a53248a8-e057-4cc5-8535-10e1ecbd3a7e") : secret "cluster-monitoring-operator-tls" not found Jul 05 13:43:03.337936 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.337847 2569 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 05 13:43:03.337936 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.337902 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls podName:2bd348fc-b188-42e1-8a0d-06d9e4c4c033 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:11.33788927 +0000 UTC m=+279.347358867 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-5q7zh" (UID: "2bd348fc-b188-42e1-8a0d-06d9e4c4c033") : secret "samples-operator-tls" not found Jul 05 13:43:03.438666 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.438637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:03.438790 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.438709 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:03.438790 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.438746 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:03.438790 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.438784 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438802 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438841 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438868 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:11.438851013 +0000 UTC m=+279.448320619 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : secret "router-metrics-certs-default" not found Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438893 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:07.438876118 +0000 UTC m=+275.448345731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : configmap "kube-rbac-proxy" not found Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438896 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438913 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle podName:6cdd610b-7966-4c3a-9648-21dbcf1828b2 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:11.43890285 +0000 UTC m=+279.448372456 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle") pod "router-default-8446587799-l6kwv" (UID: "6cdd610b-7966-4c3a-9648-21dbcf1828b2") : configmap references non-existent config key: service-ca.crt Jul 05 13:43:03.438937 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:03.438931 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:07.43892045 +0000 UTC m=+275.448390062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : secret "insights-runtime-extractor-tls" not found Jul 05 13:43:03.724136 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.724101 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" event={"ID":"70591d88-9268-44bc-bcba-453202d409ba","Type":"ContainerStarted","Data":"55cfdd051c4f05badca61aea3644b43e51d27c5aacc66b140a253ca9c7939ea8"} Jul 05 13:43:03.724285 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.724194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" event={"ID":"70591d88-9268-44bc-bcba-453202d409ba","Type":"ContainerStarted","Data":"73143d20b35dbadb25801cfa8f73e2ecfa9c8a2c597726fb2c0d270644a30aea"} Jul 05 13:43:03.725403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.725330 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jlm49_3730e0ef-cd9d-401d-ab49-9261e28e14b5/serve-healthcheck-canary/0.log" Jul 05 13:43:03.745379 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:03.745332 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-wmd89" podStartSLOduration=1.621501359 podStartE2EDuration="2.745319828s" totalImagesPullingTime="1.123818469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:43:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:03.743950543 +0000 UTC m=+271.753420174" watchObservedRunningTime="2026-07-05 13:43:03.745319828 +0000 UTC m=+271.754789448" Jul 05 13:43:07.468941 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:07.468894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:07.469305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:07.468961 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:07.469305 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:07.469002 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 05 13:43:07.469305 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:07.469066 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 05 13:43:07.469305 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:07.469079 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:15.469060652 +0000 UTC m=+283.478530270 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : configmap "kube-rbac-proxy" not found Jul 05 13:43:07.469305 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:07.469103 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls podName:78c320ec-5b42-4290-b828-cf373261e157 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:15.469093104 +0000 UTC m=+283.478562707 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qv25g" (UID: "78c320ec-5b42-4290-b828-cf373261e157") : secret "insights-runtime-extractor-tls" not found Jul 05 13:43:11.400637 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.400598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:43:11.401153 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.400659 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:43:11.401153 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:11.400740 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 05 13:43:11.401153 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:11.400804 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls podName:a53248a8-e057-4cc5-8535-10e1ecbd3a7e nodeName:}" failed. No retries permitted until 2026-07-05 13:43:27.40078811 +0000 UTC m=+295.410257708 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-n6vq6" (UID: "a53248a8-e057-4cc5-8535-10e1ecbd3a7e") : secret "cluster-monitoring-operator-tls" not found Jul 05 13:43:11.402967 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.402950 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/2bd348fc-b188-42e1-8a0d-06d9e4c4c033-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-5q7zh\" (UID: \"2bd348fc-b188-42e1-8a0d-06d9e4c4c033\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:43:11.501962 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.501939 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:11.502098 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.502034 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:11.502563 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.502543 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6cdd610b-7966-4c3a-9648-21dbcf1828b2-service-ca-bundle\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:11.504148 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.504127 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6cdd610b-7966-4c3a-9648-21dbcf1828b2-metrics-certs\") pod \"router-default-8446587799-l6kwv\" (UID: \"6cdd610b-7966-4c3a-9648-21dbcf1828b2\") " pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:11.516008 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.515991 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" Jul 05 13:43:11.617513 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.617486 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:11.638307 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.638279 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh"] Jul 05 13:43:11.745650 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.745625 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8446587799-l6kwv"] Jul 05 13:43:11.748610 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:11.748584 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cdd610b_7966_4c3a_9648_21dbcf1828b2.slice/crio-da93be74a8b01750d5e5e59e3c4f813baf93a469afd14ccf79a6123dcc81c878 WatchSource:0}: Error finding container da93be74a8b01750d5e5e59e3c4f813baf93a469afd14ccf79a6123dcc81c878: Status 404 returned error can't find the container with id da93be74a8b01750d5e5e59e3c4f813baf93a469afd14ccf79a6123dcc81c878 Jul 05 13:43:11.750152 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:11.750129 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" event={"ID":"2bd348fc-b188-42e1-8a0d-06d9e4c4c033","Type":"ContainerStarted","Data":"a22c8b0d479e6cf4fe7b7c498eb440b0aac8f096a62dd0191ebb01f2ac788ba3"} Jul 05 13:43:12.753977 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:12.753935 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8446587799-l6kwv" event={"ID":"6cdd610b-7966-4c3a-9648-21dbcf1828b2","Type":"ContainerStarted","Data":"c550901171fd8d6c8592adc6ca526d231ba9c8bfac360994a3378a8c2729875f"} Jul 05 13:43:12.754460 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:12.753991 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8446587799-l6kwv" event={"ID":"6cdd610b-7966-4c3a-9648-21dbcf1828b2","Type":"ContainerStarted","Data":"da93be74a8b01750d5e5e59e3c4f813baf93a469afd14ccf79a6123dcc81c878"} Jul 05 13:43:12.774905 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:12.774850 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8446587799-l6kwv" podStartSLOduration=17.774832318 podStartE2EDuration="17.774832318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:42:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:12.77374974 +0000 UTC m=+280.783219602" watchObservedRunningTime="2026-07-05 13:43:12.774832318 +0000 UTC m=+280.784301951" Jul 05 13:43:13.617876 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.617801 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:13.620118 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.620096 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:13.759020 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.758981 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" event={"ID":"2bd348fc-b188-42e1-8a0d-06d9e4c4c033","Type":"ContainerStarted","Data":"847052d2153eb44beb79ab85da14e45114b1fcb971030787bb75a9abac7441e3"} Jul 05 13:43:13.759402 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.759043 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" event={"ID":"2bd348fc-b188-42e1-8a0d-06d9e4c4c033","Type":"ContainerStarted","Data":"178d2e56ef3ce5894e928323667df80355960aa56b000e0f932fa43ee0f53334"} Jul 05 13:43:13.759402 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.759286 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:13.760300 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.760282 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8446587799-l6kwv" Jul 05 13:43:13.776530 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:13.776487 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-5q7zh" podStartSLOduration=17.248491356 podStartE2EDuration="18.776473254s" totalImagesPullingTime="1.527981898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:13.775535882 +0000 UTC m=+281.785005524" watchObservedRunningTime="2026-07-05 13:43:13.776473254 +0000 UTC m=+281.785942873" Jul 05 13:43:14.449614 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:14.449578 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" podUID="e45a90da-f828-44b2-9627-eb68a5f8e4c8" Jul 05 13:43:14.449614 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:14.449599 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" podUID="73359273-7ce0-4963-8590-b4e07880c348" Jul 05 13:43:14.449614 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:14.449577 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-lm677" podUID="8e146763-8e15-4664-b91d-b527657be7bb" Jul 05 13:43:14.761671 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:14.761580 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:43:14.761671 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:14.761605 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:43:14.762134 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:14.761706 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lm677" Jul 05 13:43:15.537326 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.537286 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:15.537537 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.537353 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:15.537917 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.537897 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/78c320ec-5b42-4290-b828-cf373261e157-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:15.539604 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.539579 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/78c320ec-5b42-4290-b828-cf373261e157-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qv25g\" (UID: \"78c320ec-5b42-4290-b828-cf373261e157\") " pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:15.554635 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.554613 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qv25g" Jul 05 13:43:15.676786 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.676767 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qv25g"] Jul 05 13:43:15.678620 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:15.678593 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c320ec_5b42_4290_b828_cf373261e157.slice/crio-da29d2a6deee09dc298f2c2b5064d64b9c9e68bda043a87af8575057c132706e WatchSource:0}: Error finding container da29d2a6deee09dc298f2c2b5064d64b9c9e68bda043a87af8575057c132706e: Status 404 returned error can't find the container with id da29d2a6deee09dc298f2c2b5064d64b9c9e68bda043a87af8575057c132706e Jul 05 13:43:15.765430 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:15.765394 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv25g" event={"ID":"78c320ec-5b42-4290-b828-cf373261e157","Type":"ContainerStarted","Data":"da29d2a6deee09dc298f2c2b5064d64b9c9e68bda043a87af8575057c132706e"} Jul 05 13:43:18.364614 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.364579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:43:18.364614 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.364618 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:43:18.365105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.364742 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:43:18.367071 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.367047 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8e146763-8e15-4664-b91d-b527657be7bb-metrics-tls\") pod \"dns-default-lm677\" (UID: \"8e146763-8e15-4664-b91d-b527657be7bb\") " pod="openshift-dns/dns-default-lm677" Jul 05 13:43:18.367178 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.367087 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e45a90da-f828-44b2-9627-eb68a5f8e4c8-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-g8fr4\" (UID: \"e45a90da-f828-44b2-9627-eb68a5f8e4c8\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:43:18.367178 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.367102 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"image-registry-5bd47dd89-tn97n\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:43:18.667824 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.667739 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-sw56z\"" Jul 05 13:43:18.667824 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.667740 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x86wj\"" Jul 05 13:43:18.667824 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.667785 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-j7rrk\"" Jul 05 13:43:18.673477 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.673429 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:43:18.673477 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.673459 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lm677" Jul 05 13:43:18.673659 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:18.673528 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" Jul 05 13:43:19.444802 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.444777 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4"] Jul 05 13:43:19.447468 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:19.447438 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45a90da_f828_44b2_9627_eb68a5f8e4c8.slice/crio-85ed0e3226111c0c52d238467e5f54638c33d04547bd2f34b8ec3110d16cb5d6 WatchSource:0}: Error finding container 85ed0e3226111c0c52d238467e5f54638c33d04547bd2f34b8ec3110d16cb5d6: Status 404 returned error can't find the container with id 85ed0e3226111c0c52d238467e5f54638c33d04547bd2f34b8ec3110d16cb5d6 Jul 05 13:43:19.666928 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.666883 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lm677"] Jul 05 13:43:19.669099 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:19.669059 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e146763_8e15_4664_b91d_b527657be7bb.slice/crio-e0c583d4786dad7219d72f29e47365c288df465a1eb281484111403a4461c037 WatchSource:0}: Error finding container e0c583d4786dad7219d72f29e47365c288df465a1eb281484111403a4461c037: Status 404 returned error can't find the container with id e0c583d4786dad7219d72f29e47365c288df465a1eb281484111403a4461c037 Jul 05 13:43:19.669495 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.669475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bd47dd89-tn97n"] Jul 05 13:43:19.671814 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:19.671784 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73359273_7ce0_4963_8590_b4e07880c348.slice/crio-3997c833f9a917bd7a743df05813394881955524e53a27b83d3801e5ab6fe2e0 WatchSource:0}: Error finding container 3997c833f9a917bd7a743df05813394881955524e53a27b83d3801e5ab6fe2e0: Status 404 returned error can't find the container with id 3997c833f9a917bd7a743df05813394881955524e53a27b83d3801e5ab6fe2e0 Jul 05 13:43:19.784398 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.784344 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" event={"ID":"73359273-7ce0-4963-8590-b4e07880c348","Type":"ContainerStarted","Data":"861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7"} Jul 05 13:43:19.784534 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.784422 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" event={"ID":"73359273-7ce0-4963-8590-b4e07880c348","Type":"ContainerStarted","Data":"3997c833f9a917bd7a743df05813394881955524e53a27b83d3801e5ab6fe2e0"} Jul 05 13:43:19.784534 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.784449 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:43:19.785640 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.785615 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" event={"ID":"e45a90da-f828-44b2-9627-eb68a5f8e4c8","Type":"ContainerStarted","Data":"85ed0e3226111c0c52d238467e5f54638c33d04547bd2f34b8ec3110d16cb5d6"} Jul 05 13:43:19.786977 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.786956 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv25g" event={"ID":"78c320ec-5b42-4290-b828-cf373261e157","Type":"ContainerStarted","Data":"570c2a88d04a6e81fb3bf204a5bcb1ca8bb73c7fba6bf1b5dbd718373181e971"} Jul 05 13:43:19.787990 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.787969 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lm677" event={"ID":"8e146763-8e15-4664-b91d-b527657be7bb","Type":"ContainerStarted","Data":"e0c583d4786dad7219d72f29e47365c288df465a1eb281484111403a4461c037"} Jul 05 13:43:19.805043 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:19.804999 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" podStartSLOduration=285.80498591 podStartE2EDuration="4m45.80498591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:19.804024613 +0000 UTC m=+287.813494257" watchObservedRunningTime="2026-07-05 13:43:19.80498591 +0000 UTC m=+287.814455530" Jul 05 13:43:20.798777 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:20.798744 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv25g" event={"ID":"78c320ec-5b42-4290-b828-cf373261e157","Type":"ContainerStarted","Data":"89ee66c3394125f2eb17f0a01fa68856627587dbf56ca106e4d9ccffdfe096ef"} Jul 05 13:43:21.799580 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:21.799541 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" event={"ID":"e45a90da-f828-44b2-9627-eb68a5f8e4c8","Type":"ContainerStarted","Data":"6295012dbc0d2372a364f974b45e0215076d6a57ffcdf7c97854626624ffbad4"} Jul 05 13:43:21.819153 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:21.819105 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-g8fr4" podStartSLOduration=260.520441888 podStartE2EDuration="4m21.819087403s" totalImagesPullingTime="1.298645515s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:39:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:21.817256989 +0000 UTC m=+289.826726620" watchObservedRunningTime="2026-07-05 13:43:21.819087403 +0000 UTC m=+289.828557083" Jul 05 13:43:23.807112 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:23.807077 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lm677" event={"ID":"8e146763-8e15-4664-b91d-b527657be7bb","Type":"ContainerStarted","Data":"60f7671f2767414e81945669c9370859ddb33385a23a3e55c77232f7febcb13f"} Jul 05 13:43:23.807570 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:23.807133 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lm677" event={"ID":"8e146763-8e15-4664-b91d-b527657be7bb","Type":"ContainerStarted","Data":"870bae5e619bfeb2b074d2f8b7addb90085f125e77c60bf202b8172fb2f281aa"} Jul 05 13:43:23.807570 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:23.807162 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lm677" Jul 05 13:43:23.808948 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:23.808926 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qv25g" event={"ID":"78c320ec-5b42-4290-b828-cf373261e157","Type":"ContainerStarted","Data":"b8be368e68188cb32ea3ee9b21e999a7e33843512fead237b6dd4b17d7ace23e"} Jul 05 13:43:23.826971 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:23.826920 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-lm677" podStartSLOduration=252.772653695 podStartE2EDuration="4m15.826906267s" totalImagesPullingTime="3.054252572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:23.82618841 +0000 UTC m=+291.835658048" watchObservedRunningTime="2026-07-05 13:43:23.826906267 +0000 UTC m=+291.836375886" Jul 05 13:43:23.848992 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:23.848952 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qv25g" podStartSLOduration=17.95904078 podStartE2EDuration="24.848937901s" totalImagesPullingTime="6.889897121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 13:42:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:23.848338581 +0000 UTC m=+291.857808214" watchObservedRunningTime="2026-07-05 13:43:23.848937901 +0000 UTC m=+291.858407520" Jul 05 13:43:25.795499 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:25.795460 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-jbfxn" podUID="1d342df4-57dc-415e-9232-a44b96de4d37" Jul 05 13:43:27.435712 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:27.435668 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:43:27.438195 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:27.438171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/a53248a8-e057-4cc5-8535-10e1ecbd3a7e-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-n6vq6\" (UID: \"a53248a8-e057-4cc5-8535-10e1ecbd3a7e\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:43:27.702034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:27.702008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" Jul 05 13:43:27.828354 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:27.828326 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6"] Jul 05 13:43:27.831485 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:27.831459 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53248a8_e057_4cc5_8535_10e1ecbd3a7e.slice/crio-016fc46dad489ee040a6662d52b7e40f00b6ee1c7a974853357c4359d0fc1dcb WatchSource:0}: Error finding container 016fc46dad489ee040a6662d52b7e40f00b6ee1c7a974853357c4359d0fc1dcb: Status 404 returned error can't find the container with id 016fc46dad489ee040a6662d52b7e40f00b6ee1c7a974853357c4359d0fc1dcb Jul 05 13:43:28.826226 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:28.826188 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" event={"ID":"a53248a8-e057-4cc5-8535-10e1ecbd3a7e","Type":"ContainerStarted","Data":"016fc46dad489ee040a6662d52b7e40f00b6ee1c7a974853357c4359d0fc1dcb"} Jul 05 13:43:29.830328 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:29.830291 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" event={"ID":"a53248a8-e057-4cc5-8535-10e1ecbd3a7e","Type":"ContainerStarted","Data":"3f90530492c8ae4ae562f14127340ad7b7536d2679a6515e4ee38f7f9afc4972"} Jul 05 13:43:29.849083 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:29.849028 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-n6vq6" podStartSLOduration=33.062402928 podStartE2EDuration="34.849012128s" totalImagesPullingTime="1.7866092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:29.848398648 +0000 UTC m=+297.857868290" watchObservedRunningTime="2026-07-05 13:43:29.849012128 +0000 UTC m=+297.858481749" Jul 05 13:43:32.667033 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:32.667009 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 05 13:43:33.814027 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:33.813997 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lm677" Jul 05 13:43:38.627818 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.627784 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74c69d7d66-jkjmk"] Jul 05 13:43:38.630827 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.630809 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.633824 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.633801 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 05 13:43:38.633919 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.633801 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 05 13:43:38.635334 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.635257 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 05 13:43:38.635334 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.635272 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8z5r9\"" Jul 05 13:43:38.635334 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.635289 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 05 13:43:38.635334 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.635308 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 05 13:43:38.635583 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.635406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 05 13:43:38.635620 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.635613 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 05 13:43:38.645511 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.645477 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 05 13:43:38.645966 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.645941 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c69d7d66-jkjmk"] Jul 05 13:43:38.678046 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.678024 2569 patch_prober.go:28] interesting pod/image-registry-5bd47dd89-tn97n container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 05 13:43:38.678154 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.678069 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" podUID="73359273-7ce0-4963-8590-b4e07880c348" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 13:43:38.715453 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldc5c\" (UniqueName: \"kubernetes.io/projected/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-kube-api-access-ldc5c\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.715530 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715473 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-serving-cert\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.715530 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-config\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.715530 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715509 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-service-ca\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.715645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715567 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-oauth-serving-cert\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.715645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-trusted-ca-bundle\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.715645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.715631 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-oauth-config\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816211 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816183 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-config\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816211 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816214 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-service-ca\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816355 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816233 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-oauth-serving-cert\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816355 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816314 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-trusted-ca-bundle\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816475 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816381 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-oauth-config\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816475 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldc5c\" (UniqueName: \"kubernetes.io/projected/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-kube-api-access-ldc5c\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.816586 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816492 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-serving-cert\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.817001 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.816976 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-service-ca\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.817114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.817045 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-config\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.817114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.817087 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-oauth-serving-cert\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.817384 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.817350 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-trusted-ca-bundle\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.818838 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.818812 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-oauth-config\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.818935 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.818919 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-serving-cert\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.826112 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.826088 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldc5c\" (UniqueName: \"kubernetes.io/projected/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-kube-api-access-ldc5c\") pod \"console-74c69d7d66-jkjmk\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:38.942080 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:38.942000 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:39.070788 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:39.070761 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74c69d7d66-jkjmk"] Jul 05 13:43:39.073295 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:39.073272 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f932b5_0dfc_47eb_80e2_b43594bf36e7.slice/crio-1f44954340a5c0522b312cd6b90488cd4103b0516fc093c0f73fd790f1700d36 WatchSource:0}: Error finding container 1f44954340a5c0522b312cd6b90488cd4103b0516fc093c0f73fd790f1700d36: Status 404 returned error can't find the container with id 1f44954340a5c0522b312cd6b90488cd4103b0516fc093c0f73fd790f1700d36 Jul 05 13:43:39.077184 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:39.076894 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 13:43:39.793246 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:39.793210 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:43:39.864772 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:39.864741 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c69d7d66-jkjmk" event={"ID":"a9f932b5-0dfc-47eb-80e2-b43594bf36e7","Type":"ContainerStarted","Data":"1f44954340a5c0522b312cd6b90488cd4103b0516fc093c0f73fd790f1700d36"} Jul 05 13:43:40.799518 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:40.799493 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:43:41.873779 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:41.873746 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c69d7d66-jkjmk" event={"ID":"a9f932b5-0dfc-47eb-80e2-b43594bf36e7","Type":"ContainerStarted","Data":"b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35"} Jul 05 13:43:41.893740 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:41.893697 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74c69d7d66-jkjmk" podStartSLOduration=1.194768552 podStartE2EDuration="3.893680536s" totalImagesPullingTime="2.698911984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:43:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:41.893400517 +0000 UTC m=+309.902870163" watchObservedRunningTime="2026-07-05 13:43:41.893680536 +0000 UTC m=+309.903150156" Jul 05 13:43:44.006031 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:44.005997 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bd47dd89-tn97n"] Jul 05 13:43:45.500102 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.500071 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg"] Jul 05 13:43:45.506273 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.506254 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.509502 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.509470 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 05 13:43:45.509599 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.509526 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 05 13:43:45.510606 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.510590 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 05 13:43:45.510682 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.510588 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-2lgwg\"" Jul 05 13:43:45.514555 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.514537 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx"] Jul 05 13:43:45.518105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.518092 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.521597 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.521462 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 05 13:43:45.521597 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.521483 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-b54kn\"" Jul 05 13:43:45.522230 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.522208 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 05 13:43:45.522326 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.522232 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 05 13:43:45.523382 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.523348 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg"] Jul 05 13:43:45.529482 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.529464 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx"] Jul 05 13:43:45.532571 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.532553 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-m455r"] Jul 05 13:43:45.536245 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.536224 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.539002 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.538986 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2nddw\"" Jul 05 13:43:45.539089 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.539030 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 05 13:43:45.539236 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.539223 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 05 13:43:45.539318 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.539256 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 05 13:43:45.576447 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.576447 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576450 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.576648 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576477 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.576648 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576529 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.576648 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.576648 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwdv\" (UniqueName: \"kubernetes.io/projected/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-api-access-hxwdv\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.576799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b12-8a35-49e8-949c-6bb5264e0812-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.576799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c767b12-8a35-49e8-949c-6bb5264e0812-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.576799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64h89\" (UniqueName: \"kubernetes.io/projected/100633fb-3a92-4178-9eb1-500bc480dbbc-kube-api-access-64h89\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.576799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.576786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100633fb-3a92-4178-9eb1-500bc480dbbc-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.677498 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677470 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-wtmp\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.677498 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-accelerators-collector-config\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677522 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hv97v\" (UniqueName: \"kubernetes.io/projected/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-kube-api-access-hv97v\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677553 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100633fb-3a92-4178-9eb1-500bc480dbbc-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-textfile\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677587 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-tls\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677607 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.677710 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-sys\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:45.677721 2569 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:45.677780 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-tls podName:100633fb-3a92-4178-9eb1-500bc480dbbc nodeName:}" failed. No retries permitted until 2026-07-05 13:43:46.177766643 +0000 UTC m=+314.187236241 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-kvghg" (UID: "100633fb-3a92-4178-9eb1-500bc480dbbc") : secret "openshift-state-metrics-tls" not found Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677810 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677833 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-root\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677869 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.677960 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-metrics-client-ca\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.678064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678003 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwdv\" (UniqueName: \"kubernetes.io/projected/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-api-access-hxwdv\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678508 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678092 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b12-8a35-49e8-949c-6bb5264e0812-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678508 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678214 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/100633fb-3a92-4178-9eb1-500bc480dbbc-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.678664 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678643 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678834 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678807 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c767b12-8a35-49e8-949c-6bb5264e0812-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678816 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c767b12-8a35-49e8-949c-6bb5264e0812-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.678920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64h89\" (UniqueName: \"kubernetes.io/projected/100633fb-3a92-4178-9eb1-500bc480dbbc-kube-api-access-64h89\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.679018 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.678914 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.679074 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.679039 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8c767b12-8a35-49e8-949c-6bb5264e0812-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.680210 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.680180 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.680295 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.680255 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.680391 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.680355 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.689164 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.689140 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwdv\" (UniqueName: \"kubernetes.io/projected/8c767b12-8a35-49e8-949c-6bb5264e0812-kube-api-access-hxwdv\") pod \"kube-state-metrics-7c9dbb8b7d-tsdgx\" (UID: \"8c767b12-8a35-49e8-949c-6bb5264e0812\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.690234 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.690214 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64h89\" (UniqueName: \"kubernetes.io/projected/100633fb-3a92-4178-9eb1-500bc480dbbc-kube-api-access-64h89\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:45.780004 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.779904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780004 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.779947 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-wtmp\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780004 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.779968 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-accelerators-collector-config\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780072 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hv97v\" (UniqueName: \"kubernetes.io/projected/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-kube-api-access-hv97v\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780118 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-textfile\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780120 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-wtmp\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-tls\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780187 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-sys\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-root\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:45.780251 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 05 13:43:45.780267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780267 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-sys\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780271 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-metrics-client-ca\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780702 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:43:45.780308 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-tls podName:e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46 nodeName:}" failed. No retries permitted until 2026-07-05 13:43:46.280289348 +0000 UTC m=+314.289758956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-tls") pod "node-exporter-m455r" (UID: "e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46") : secret "node-exporter-tls" not found Jul 05 13:43:45.780702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780324 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-root\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780603 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-accelerators-collector-config\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780653 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-textfile\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.780956 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.780718 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-metrics-client-ca\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.782359 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.782336 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.789439 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.789423 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hv97v\" (UniqueName: \"kubernetes.io/projected/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-kube-api-access-hv97v\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:45.827066 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.827039 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" Jul 05 13:43:45.960067 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:45.960031 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx"] Jul 05 13:43:45.963608 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:45.963583 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c767b12_8a35_49e8_949c_6bb5264e0812.slice/crio-ce56e3f8475cadfada601b21af7df67e94d5bc1dbc651fa0c6876d98b52e9321 WatchSource:0}: Error finding container ce56e3f8475cadfada601b21af7df67e94d5bc1dbc651fa0c6876d98b52e9321: Status 404 returned error can't find the container with id ce56e3f8475cadfada601b21af7df67e94d5bc1dbc651fa0c6876d98b52e9321 Jul 05 13:43:46.184602 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.184528 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:46.186642 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.186622 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/100633fb-3a92-4178-9eb1-500bc480dbbc-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-kvghg\" (UID: \"100633fb-3a92-4178-9eb1-500bc480dbbc\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:46.285478 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.285441 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-tls\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:46.287612 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.287592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46-node-exporter-tls\") pod \"node-exporter-m455r\" (UID: \"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46\") " pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:46.414884 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.414851 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" Jul 05 13:43:46.444477 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.444405 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-m455r" Jul 05 13:43:46.456719 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:46.456690 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f6fda1_1db9_4dfd_bc4a_5c8023ba3e46.slice/crio-b430ee396a47f9ea7fa8294c857c4a4878b4ab1deae8fab8c179066e4aa90bea WatchSource:0}: Error finding container b430ee396a47f9ea7fa8294c857c4a4878b4ab1deae8fab8c179066e4aa90bea: Status 404 returned error can't find the container with id b430ee396a47f9ea7fa8294c857c4a4878b4ab1deae8fab8c179066e4aa90bea Jul 05 13:43:46.544883 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.544859 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg"] Jul 05 13:43:46.546997 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:46.546973 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod100633fb_3a92_4178_9eb1_500bc480dbbc.slice/crio-7057ea8bf845b39ead784e67ae524339f3e59617112d77486a3bb1f9fb7eb8b2 WatchSource:0}: Error finding container 7057ea8bf845b39ead784e67ae524339f3e59617112d77486a3bb1f9fb7eb8b2: Status 404 returned error can't find the container with id 7057ea8bf845b39ead784e67ae524339f3e59617112d77486a3bb1f9fb7eb8b2 Jul 05 13:43:46.895750 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.895714 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" event={"ID":"100633fb-3a92-4178-9eb1-500bc480dbbc","Type":"ContainerStarted","Data":"61b7573af2725df4ac13d8c2aa2149a19756d741ac1ac161eb05b65a62a53a14"} Jul 05 13:43:46.895923 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.895783 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" event={"ID":"100633fb-3a92-4178-9eb1-500bc480dbbc","Type":"ContainerStarted","Data":"59d97864699586795744f19e75460603bf0ed741ec467a3a88b841fbd7745583"} Jul 05 13:43:46.895923 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.895810 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" event={"ID":"100633fb-3a92-4178-9eb1-500bc480dbbc","Type":"ContainerStarted","Data":"7057ea8bf845b39ead784e67ae524339f3e59617112d77486a3bb1f9fb7eb8b2"} Jul 05 13:43:46.896975 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.896947 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m455r" event={"ID":"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46","Type":"ContainerStarted","Data":"b430ee396a47f9ea7fa8294c857c4a4878b4ab1deae8fab8c179066e4aa90bea"} Jul 05 13:43:46.898164 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:46.898127 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" event={"ID":"8c767b12-8a35-49e8-949c-6bb5264e0812","Type":"ContainerStarted","Data":"ce56e3f8475cadfada601b21af7df67e94d5bc1dbc651fa0c6876d98b52e9321"} Jul 05 13:43:47.905511 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:47.905394 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" event={"ID":"8c767b12-8a35-49e8-949c-6bb5264e0812","Type":"ContainerStarted","Data":"abbdcfbb179884d979face05dcba010206d2ba4e1ccf5f891ce10203a1ba06fe"} Jul 05 13:43:47.905511 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:47.905460 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" event={"ID":"8c767b12-8a35-49e8-949c-6bb5264e0812","Type":"ContainerStarted","Data":"010d69a1ce261c8db0d1c9c1cf55d03bcecbee6118dc5e69821888bf8e22c1c6"} Jul 05 13:43:47.905511 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:47.905484 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" event={"ID":"8c767b12-8a35-49e8-949c-6bb5264e0812","Type":"ContainerStarted","Data":"d629052da52cc35843c2207ab305bf0126b4398b7bdc3e5ca38692d23a213696"} Jul 05 13:43:47.908207 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:47.908157 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46" containerID="7fdd6a59189cda42fb5ece80bbc3959981a33c07cb6f826e3b0330d2b3e75c28" exitCode=0 Jul 05 13:43:47.908332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:47.908209 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m455r" event={"ID":"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46","Type":"ContainerDied","Data":"7fdd6a59189cda42fb5ece80bbc3959981a33c07cb6f826e3b0330d2b3e75c28"} Jul 05 13:43:47.927561 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:47.927514 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-tsdgx" podStartSLOduration=1.3701153019999999 podStartE2EDuration="2.927495077s" totalImagesPullingTime="1.557379775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:47.925749707 +0000 UTC m=+315.935219342" watchObservedRunningTime="2026-07-05 13:43:47.927495077 +0000 UTC m=+315.936964700" Jul 05 13:43:48.913883 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:48.913843 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" event={"ID":"100633fb-3a92-4178-9eb1-500bc480dbbc","Type":"ContainerStarted","Data":"88cc655d14eafd3fb77c5d67c81b21204c0e0be0be3d54d264b4689c7a73ea92"} Jul 05 13:43:48.915884 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:48.915858 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m455r" event={"ID":"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46","Type":"ContainerStarted","Data":"0d1d013f9ad50a1ca28e0b1ba1803214f197d361d5d01bde01f5809d1ec4025f"} Jul 05 13:43:48.915981 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:48.915901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-m455r" event={"ID":"e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46","Type":"ContainerStarted","Data":"136c3ccc8f13dde99c355f2a5b3c883ee7f74b0307b7acf8b208fb4596a477ac"} Jul 05 13:43:48.942098 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:48.942066 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:48.942195 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:48.942114 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:48.946774 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:48.946756 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:49.014194 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:49.014151 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-m455r" podStartSLOduration=2.839608485 podStartE2EDuration="4.014139276s" totalImagesPullingTime="1.06757789s" totalInitContainerRuntime="106.952901ms" isStatefulPod=true podCreationTimestamp="2026-07-05 13:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:49.013860854 +0000 UTC m=+317.023330467" watchObservedRunningTime="2026-07-05 13:43:49.014139276 +0000 UTC m=+317.023608895" Jul 05 13:43:49.014930 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:49.014902 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-kvghg" podStartSLOduration=2.6547732760000002 podStartE2EDuration="4.014896022s" totalImagesPullingTime="1.360122746s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:43:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:43:48.961504922 +0000 UTC m=+316.970974543" watchObservedRunningTime="2026-07-05 13:43:49.014896022 +0000 UTC m=+317.024365641" Jul 05 13:43:49.923891 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:49.923859 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:43:51.693868 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.693838 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:43:51.880083 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.880047 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:43:51.880232 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.880172 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:51.883651 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.883628 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 05 13:43:51.883771 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.883702 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 05 13:43:51.883955 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.883937 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 05 13:43:51.884044 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884023 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 05 13:43:51.884044 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884034 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 05 13:43:51.884208 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884191 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 05 13:43:51.884271 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884256 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kqkrr\"" Jul 05 13:43:51.884310 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884258 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 05 13:43:51.884310 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884288 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 05 13:43:51.884615 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884597 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 05 13:43:51.884671 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.884629 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 05 13:43:51.885119 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.885104 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 05 13:43:51.885177 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.885128 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-111ohm7n1mmrs\"" Jul 05 13:43:51.885219 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.885127 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 05 13:43:51.885447 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.885427 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 05 13:43:51.888284 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.888264 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 05 13:43:51.890528 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:51.890508 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 05 13:43:52.036598 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.036598 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.036825 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036637 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.036825 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036679 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.036825 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036773 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.036825 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036816 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036850 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036888 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-config-out\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-config\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036976 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.036992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-web-config\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037007 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037073 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh2kk\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-kube-api-access-hh2kk\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037147 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.037329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.037238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138233 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138207 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-config-out\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138268 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138293 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-config\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-web-config\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138702 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138405 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138756 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138741 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.138901 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139025 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138918 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh2kk\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-kube-api-access-hh2kk\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139025 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.138976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139025 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139015 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139188 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139052 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139188 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139188 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139084 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139188 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139122 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139188 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139207 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139217 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139300 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139328 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.139743 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.139723 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.140253 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.140229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.140974 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.140945 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-config\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.142114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.141125 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-config-out\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.142114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.141449 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.142114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.141522 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-web-config\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.142114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.141764 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.142114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.141845 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.142114 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.142020 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.148815 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.143084 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.148815 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.145037 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.148815 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.148749 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.149007 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.148908 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.149007 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.148918 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.149007 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.148908 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.149160 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.149095 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.150961 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.150943 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh2kk\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-kube-api-access-hh2kk\") pod \"prometheus-k8s-0\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.190031 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.190011 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:43:52.331332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.331215 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:43:52.333319 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:43:52.333284 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1bd76c4_f260_45b5_a232_df186b56bd71.slice/crio-c1cfc4b46b376962fddb584041cc9405213c58828c23715aa728dbab2c2d53a1 WatchSource:0}: Error finding container c1cfc4b46b376962fddb584041cc9405213c58828c23715aa728dbab2c2d53a1: Status 404 returned error can't find the container with id c1cfc4b46b376962fddb584041cc9405213c58828c23715aa728dbab2c2d53a1 Jul 05 13:43:52.932423 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:52.932384 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"c1cfc4b46b376962fddb584041cc9405213c58828c23715aa728dbab2c2d53a1"} Jul 05 13:43:53.937017 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:53.936940 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="c555be320bd1429726a41727db5c6e66f1673b83b9b2df7a412b893247ce10e2" exitCode=0 Jul 05 13:43:53.937403 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:53.937024 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"c555be320bd1429726a41727db5c6e66f1673b83b9b2df7a412b893247ce10e2"} Jul 05 13:43:57.956056 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:57.956022 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"353740a09da822960ce2d7aeebdedef8de1b18bd9ce41e479df6c4d9f2166215"} Jul 05 13:43:57.956425 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:57.956090 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"67d47b344e931959a61660f028617c47ce8c9c8a305fe1ef27fb64eb33c0e772"} Jul 05 13:43:58.967835 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:58.967808 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"d63f6dc734487ead525480da9ffa93d437178e5745bae2ec61364e1e9bb10462"} Jul 05 13:43:58.968178 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:58.967901 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"c151811ceb7d8664c1b89879696360be221df6c604c96e933e7159676c63e6b1"} Jul 05 13:43:58.968178 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:58.967928 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"cf4bc044c8256dcb3407debe1cd939cc94c2fd3c668ace66f8bdf33712d220b4"} Jul 05 13:43:59.974579 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:43:59.974546 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerStarted","Data":"9f9d1d34d4a2f8ee2d49fededdd39d771ec0141d4d64fccddf8d84192755e323"} Jul 05 13:44:00.019669 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:00.019605 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.075441937 podStartE2EDuration="9.019587146s" totalImagesPullingTime="5.925597426s" totalInitContainerRuntime="18.547783ms" isStatefulPod=false podCreationTimestamp="2026-07-05 13:43:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:44:00.017700594 +0000 UTC m=+328.027170237" watchObservedRunningTime="2026-07-05 13:44:00.019587146 +0000 UTC m=+328.029056786" Jul 05 13:44:02.190338 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:02.190301 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:44:07.674161 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:07.674119 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c69d7d66-jkjmk"] Jul 05 13:44:09.032598 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.032554 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" podUID="73359273-7ce0-4963-8590-b4e07880c348" containerName="registry" containerID="cri-o://861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7" gracePeriod=30 Jul 05 13:44:09.279250 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.279228 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:44:09.284068 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284024 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tkxwb\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-kube-api-access-tkxwb\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284167 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284067 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-image-registry-private-configuration\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284167 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284115 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-registry-certificates\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284167 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284133 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-bound-sa-token\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284167 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284153 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284175 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-trusted-ca\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284208 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73359273-7ce0-4963-8590-b4e07880c348-ca-trust-extracted\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284396 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284248 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-installation-pull-secrets\") pod \"73359273-7ce0-4963-8590-b4e07880c348\" (UID: \"73359273-7ce0-4963-8590-b4e07880c348\") " Jul 05 13:44:09.284799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284768 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:44:09.284893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.284790 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:44:09.286570 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.286518 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-kube-api-access-tkxwb" (OuterVolumeSpecName: "kube-api-access-tkxwb") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "kube-api-access-tkxwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:44:09.286664 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.286622 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:44:09.286877 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.286847 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:44:09.286978 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.286928 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:44:09.287092 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.287075 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:44:09.293025 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.293002 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73359273-7ce0-4963-8590-b4e07880c348-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "73359273-7ce0-4963-8590-b4e07880c348" (UID: "73359273-7ce0-4963-8590-b4e07880c348"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 13:44:09.385275 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385247 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-bound-sa-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385275 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385271 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-registry-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385275 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385280 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-trusted-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385489 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385289 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/73359273-7ce0-4963-8590-b4e07880c348-ca-trust-extracted\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385489 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385298 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-installation-pull-secrets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385489 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385307 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkxwb\" (UniqueName: \"kubernetes.io/projected/73359273-7ce0-4963-8590-b4e07880c348-kube-api-access-tkxwb\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385489 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385315 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/73359273-7ce0-4963-8590-b4e07880c348-image-registry-private-configuration\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:09.385489 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:09.385324 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/73359273-7ce0-4963-8590-b4e07880c348-registry-certificates\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:10.015792 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.015759 2569 generic.go:356] "Generic (PLEG): container finished" podID="73359273-7ce0-4963-8590-b4e07880c348" containerID="861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7" exitCode=0 Jul 05 13:44:10.015931 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.015803 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" Jul 05 13:44:10.015931 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.015845 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" event={"ID":"73359273-7ce0-4963-8590-b4e07880c348","Type":"ContainerDied","Data":"861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7"} Jul 05 13:44:10.015931 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.015893 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bd47dd89-tn97n" event={"ID":"73359273-7ce0-4963-8590-b4e07880c348","Type":"ContainerDied","Data":"3997c833f9a917bd7a743df05813394881955524e53a27b83d3801e5ab6fe2e0"} Jul 05 13:44:10.015931 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.015913 2569 scope.go:117] "RemoveContainer" containerID="861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7" Jul 05 13:44:10.024966 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.024934 2569 scope.go:117] "RemoveContainer" containerID="861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7" Jul 05 13:44:10.025211 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:44:10.025188 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7\": container with ID starting with 861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7 not found: ID does not exist" containerID="861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7" Jul 05 13:44:10.025272 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.025218 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7"} err="failed to get container status \"861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7\": rpc error: code = NotFound desc = could not find container \"861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7\": container with ID starting with 861fea719a314660032987c4917d34270da4dd23960bc471a5a38b17ae5c6fd7 not found: ID does not exist" Jul 05 13:44:10.037902 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.037880 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5bd47dd89-tn97n"] Jul 05 13:44:10.041977 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.041959 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5bd47dd89-tn97n"] Jul 05 13:44:10.797709 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:10.797679 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73359273-7ce0-4963-8590-b4e07880c348" path="/var/lib/kubelet/pods/73359273-7ce0-4963-8590-b4e07880c348/volumes" Jul 05 13:44:30.093142 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:30.093105 2569 generic.go:356] "Generic (PLEG): container finished" podID="19560aa3-2a88-468c-aa1c-0320e499a8ff" containerID="8c89a42f8a3855063ef7834d2c88ff9f3ba68bfe3fba3b717444326bc5c632d0" exitCode=0 Jul 05 13:44:30.093650 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:30.093184 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" event={"ID":"19560aa3-2a88-468c-aa1c-0320e499a8ff","Type":"ContainerDied","Data":"8c89a42f8a3855063ef7834d2c88ff9f3ba68bfe3fba3b717444326bc5c632d0"} Jul 05 13:44:30.093650 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:30.093503 2569 scope.go:117] "RemoveContainer" containerID="8c89a42f8a3855063ef7834d2c88ff9f3ba68bfe3fba3b717444326bc5c632d0" Jul 05 13:44:30.094759 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:30.094730 2569 generic.go:356] "Generic (PLEG): container finished" podID="054b55e9-2ef5-4320-ad4c-c25d2c221495" containerID="cdc5c25a68dfa7f1f3cad4947a1ade7897adef10e671fb2100eb6f2bbcbfd217" exitCode=0 Jul 05 13:44:30.094818 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:30.094763 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" event={"ID":"054b55e9-2ef5-4320-ad4c-c25d2c221495","Type":"ContainerDied","Data":"cdc5c25a68dfa7f1f3cad4947a1ade7897adef10e671fb2100eb6f2bbcbfd217"} Jul 05 13:44:30.095030 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:30.095015 2569 scope.go:117] "RemoveContainer" containerID="cdc5c25a68dfa7f1f3cad4947a1ade7897adef10e671fb2100eb6f2bbcbfd217" Jul 05 13:44:31.100247 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:31.100211 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-4d8vb" event={"ID":"19560aa3-2a88-468c-aa1c-0320e499a8ff","Type":"ContainerStarted","Data":"c333aa0f95ec8bdd6394f61364d36d5edbed541bed48be3ba818ada2c3528829"} Jul 05 13:44:31.101948 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:31.101927 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-xrkbr" event={"ID":"054b55e9-2ef5-4320-ad4c-c25d2c221495","Type":"ContainerStarted","Data":"b65065a9a3af678b19b9ba4e7e332bc90bff7feadefd828d4c22d114399eaab7"} Jul 05 13:44:32.695040 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:32.694991 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-74c69d7d66-jkjmk" podUID="a9f932b5-0dfc-47eb-80e2-b43594bf36e7" containerName="console" containerID="cri-o://b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35" gracePeriod=15 Jul 05 13:44:32.939956 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:32.939934 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:44:33.086746 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086716 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-oauth-serving-cert\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.086920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086763 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-service-ca\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.086920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086787 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldc5c\" (UniqueName: \"kubernetes.io/projected/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-kube-api-access-ldc5c\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.086920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086813 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-trusted-ca-bundle\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.086920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086906 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-serving-cert\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.087140 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086944 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-oauth-config\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.087140 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.086974 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-config\") pod \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\" (UID: \"a9f932b5-0dfc-47eb-80e2-b43594bf36e7\") " Jul 05 13:44:33.087236 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.087166 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:44:33.087236 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.087214 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:44:33.087336 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.087242 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-service-ca" (OuterVolumeSpecName: "service-ca") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:44:33.087410 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.087359 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-trusted-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:33.087410 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.087398 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-oauth-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:33.087516 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.087459 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-config" (OuterVolumeSpecName: "console-config") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:44:33.089252 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.089226 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:44:33.089405 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.089386 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:44:33.089469 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.089410 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-kube-api-access-ldc5c" (OuterVolumeSpecName: "kube-api-access-ldc5c") pod "a9f932b5-0dfc-47eb-80e2-b43594bf36e7" (UID: "a9f932b5-0dfc-47eb-80e2-b43594bf36e7"). InnerVolumeSpecName "kube-api-access-ldc5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:44:33.110781 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.110755 2569 generic.go:356] "Generic (PLEG): container finished" podID="a9f932b5-0dfc-47eb-80e2-b43594bf36e7" containerID="b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35" exitCode=0 Jul 05 13:44:33.110900 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.110809 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c69d7d66-jkjmk" event={"ID":"a9f932b5-0dfc-47eb-80e2-b43594bf36e7","Type":"ContainerDied","Data":"b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35"} Jul 05 13:44:33.110900 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.110813 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74c69d7d66-jkjmk" Jul 05 13:44:33.110900 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.110846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74c69d7d66-jkjmk" event={"ID":"a9f932b5-0dfc-47eb-80e2-b43594bf36e7","Type":"ContainerDied","Data":"1f44954340a5c0522b312cd6b90488cd4103b0516fc093c0f73fd790f1700d36"} Jul 05 13:44:33.110900 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.110865 2569 scope.go:117] "RemoveContainer" containerID="b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35" Jul 05 13:44:33.119452 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.119436 2569 scope.go:117] "RemoveContainer" containerID="b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35" Jul 05 13:44:33.119693 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:44:33.119675 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35\": container with ID starting with b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35 not found: ID does not exist" containerID="b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35" Jul 05 13:44:33.119754 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.119705 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35"} err="failed to get container status \"b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35\": rpc error: code = NotFound desc = could not find container \"b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35\": container with ID starting with b22825cc94cebf35287f2888326f040848c2e651ae4ec3f36fba77f498b4fd35 not found: ID does not exist" Jul 05 13:44:33.133050 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.133028 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74c69d7d66-jkjmk"] Jul 05 13:44:33.140705 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.140682 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74c69d7d66-jkjmk"] Jul 05 13:44:33.188781 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.188730 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-service-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:33.188781 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.188751 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldc5c\" (UniqueName: \"kubernetes.io/projected/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-kube-api-access-ldc5c\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:33.188781 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.188762 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-serving-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:33.188781 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.188771 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-oauth-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:33.188940 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:33.188789 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9f932b5-0dfc-47eb-80e2-b43594bf36e7-console-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:44:34.798428 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:34.798393 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9f932b5-0dfc-47eb-80e2-b43594bf36e7" path="/var/lib/kubelet/pods/a9f932b5-0dfc-47eb-80e2-b43594bf36e7/volumes" Jul 05 13:44:46.698138 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:46.698106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:44:46.700324 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:46.700293 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1d342df4-57dc-415e-9232-a44b96de4d37-metrics-certs\") pod \"network-metrics-daemon-jbfxn\" (UID: \"1d342df4-57dc-415e-9232-a44b96de4d37\") " pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:44:46.996971 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:46.996891 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fvggw\"" Jul 05 13:44:47.003788 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:47.003757 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jbfxn" Jul 05 13:44:47.133334 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:47.133304 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jbfxn"] Jul 05 13:44:47.136787 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:44:47.136760 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d342df4_57dc_415e_9232_a44b96de4d37.slice/crio-90ffe1b1c6e5404b129c1410c1405feaaf6112652c36cfe2d8beb5cc280d7528 WatchSource:0}: Error finding container 90ffe1b1c6e5404b129c1410c1405feaaf6112652c36cfe2d8beb5cc280d7528: Status 404 returned error can't find the container with id 90ffe1b1c6e5404b129c1410c1405feaaf6112652c36cfe2d8beb5cc280d7528 Jul 05 13:44:47.172717 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:47.172690 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbfxn" event={"ID":"1d342df4-57dc-415e-9232-a44b96de4d37","Type":"ContainerStarted","Data":"90ffe1b1c6e5404b129c1410c1405feaaf6112652c36cfe2d8beb5cc280d7528"} Jul 05 13:44:48.178211 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:48.178184 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbfxn" event={"ID":"1d342df4-57dc-415e-9232-a44b96de4d37","Type":"ContainerStarted","Data":"53b69341b09dea6717bd7b68c8c7b22392b3319d36816074ff53da5d18151a82"} Jul 05 13:44:49.183782 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:49.183745 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jbfxn" event={"ID":"1d342df4-57dc-415e-9232-a44b96de4d37","Type":"ContainerStarted","Data":"85e46c75726d963c083d0de29fb3bcb8c0ce2de8c857018e1673b6ff20138120"} Jul 05 13:44:49.201115 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:49.201064 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jbfxn" podStartSLOduration=375.316735608 podStartE2EDuration="6m16.201047857s" totalImagesPullingTime="884.312249ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:44:49.200003227 +0000 UTC m=+377.209472870" watchObservedRunningTime="2026-07-05 13:44:49.201047857 +0000 UTC m=+377.210517543" Jul 05 13:44:52.190940 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:52.190903 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:44:52.205809 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:52.205785 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:44:53.213446 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:44:53.213421 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:10.071872 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.071829 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:45:10.072429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.072131 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="prometheus" containerID="cri-o://67d47b344e931959a61660f028617c47ce8c9c8a305fe1ef27fb64eb33c0e772" gracePeriod=600 Jul 05 13:45:10.072429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.072159 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy" containerID="cri-o://d63f6dc734487ead525480da9ffa93d437178e5745bae2ec61364e1e9bb10462" gracePeriod=600 Jul 05 13:45:10.072429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.072193 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="thanos-sidecar" containerID="cri-o://cf4bc044c8256dcb3407debe1cd939cc94c2fd3c668ace66f8bdf33712d220b4" gracePeriod=600 Jul 05 13:45:10.072429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.072203 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9f9d1d34d4a2f8ee2d49fededdd39d771ec0141d4d64fccddf8d84192755e323" gracePeriod=600 Jul 05 13:45:10.072429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.072231 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="config-reloader" containerID="cri-o://353740a09da822960ce2d7aeebdedef8de1b18bd9ce41e479df6c4d9f2166215" gracePeriod=600 Jul 05 13:45:10.072429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.072212 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-web" containerID="cri-o://c151811ceb7d8664c1b89879696360be221df6c604c96e933e7159676c63e6b1" gracePeriod=600 Jul 05 13:45:10.265913 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265884 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="9f9d1d34d4a2f8ee2d49fededdd39d771ec0141d4d64fccddf8d84192755e323" exitCode=0 Jul 05 13:45:10.265913 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265910 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="d63f6dc734487ead525480da9ffa93d437178e5745bae2ec61364e1e9bb10462" exitCode=0 Jul 05 13:45:10.265913 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265916 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="c151811ceb7d8664c1b89879696360be221df6c604c96e933e7159676c63e6b1" exitCode=0 Jul 05 13:45:10.265913 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265923 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="cf4bc044c8256dcb3407debe1cd939cc94c2fd3c668ace66f8bdf33712d220b4" exitCode=0 Jul 05 13:45:10.265913 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265929 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="353740a09da822960ce2d7aeebdedef8de1b18bd9ce41e479df6c4d9f2166215" exitCode=0 Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265934 2569 generic.go:356] "Generic (PLEG): container finished" podID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerID="67d47b344e931959a61660f028617c47ce8c9c8a305fe1ef27fb64eb33c0e772" exitCode=0 Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.265954 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"9f9d1d34d4a2f8ee2d49fededdd39d771ec0141d4d64fccddf8d84192755e323"} Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.266020 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"d63f6dc734487ead525480da9ffa93d437178e5745bae2ec61364e1e9bb10462"} Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.266046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"c151811ceb7d8664c1b89879696360be221df6c604c96e933e7159676c63e6b1"} Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.266069 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"cf4bc044c8256dcb3407debe1cd939cc94c2fd3c668ace66f8bdf33712d220b4"} Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.266094 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"353740a09da822960ce2d7aeebdedef8de1b18bd9ce41e479df6c4d9f2166215"} Jul 05 13:45:10.266159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.266117 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"67d47b344e931959a61660f028617c47ce8c9c8a305fe1ef27fb64eb33c0e772"} Jul 05 13:45:10.315284 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.315264 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:10.379415 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379326 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-0\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379415 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379356 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-1\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379415 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379395 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379659 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379549 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-web-config\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379659 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379585 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-config-out\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379659 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh2kk\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-kube-api-access-hh2kk\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379659 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-thanos-prometheus-http-client-file\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379667 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-config\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379691 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379721 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-metrics-client-ca\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379723 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379747 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-serving-certs-ca-bundle\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379774 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-tls-assets\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379816 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-2\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.379875 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379841 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-metrics-client-certs\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-db\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379918 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-trusted-ca-bundle\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379949 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-kubelet-serving-ca-bundle\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.379999 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-grpc-tls\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.380044 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-tls\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.380074 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-kube-rbac-proxy\") pod \"e1bd76c4-f260-45b5-a232-df186b56bd71\" (UID: \"e1bd76c4-f260-45b5-a232-df186b56bd71\") " Jul 05 13:45:10.380594 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.380344 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.380914 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.380885 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.382053 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.381405 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.382053 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.381773 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.382887 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.382857 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.383169 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.383146 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-config" (OuterVolumeSpecName: "config") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.383509 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.383477 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.383812 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.383786 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-kube-api-access-hh2kk" (OuterVolumeSpecName: "kube-api-access-hh2kk") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "kube-api-access-hh2kk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:45:10.383969 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.383945 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 13:45:10.384034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.383991 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 13:45:10.384197 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.384166 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.384774 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.384399 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.384774 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.384694 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.384919 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.384896 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-config-out" (OuterVolumeSpecName: "config-out") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 13:45:10.385034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.385009 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.385141 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.385109 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.386239 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.386208 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.386320 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.386244 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.386360 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.386316 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:45:10.393523 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.393504 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-web-config" (OuterVolumeSpecName: "web-config") pod "e1bd76c4-f260-45b5-a232-df186b56bd71" (UID: "e1bd76c4-f260-45b5-a232-df186b56bd71"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:45:10.481033 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481013 2569 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-grpc-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481033 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481032 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481041 2569 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-kube-rbac-proxy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481050 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481060 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481068 2569 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-web-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481076 2569 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-config-out\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481083 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh2kk\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-kube-api-access-hh2kk\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481091 2569 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-thanos-prometheus-http-client-file\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481099 2569 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-config\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481107 2569 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481116 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-metrics-client-ca\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481124 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481132 2569 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e1bd76c4-f260-45b5-a232-df186b56bd71-tls-assets\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481140 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481145 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481148 2569 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e1bd76c4-f260-45b5-a232-df186b56bd71-secret-metrics-client-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481572 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481157 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-k8s-db\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481572 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481165 2569 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-prometheus-trusted-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:10.481572 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:10.481174 2569 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1bd76c4-f260-45b5-a232-df186b56bd71-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:45:11.272417 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.272385 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e1bd76c4-f260-45b5-a232-df186b56bd71","Type":"ContainerDied","Data":"c1cfc4b46b376962fddb584041cc9405213c58828c23715aa728dbab2c2d53a1"} Jul 05 13:45:11.272777 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.272457 2569 scope.go:117] "RemoveContainer" containerID="9f9d1d34d4a2f8ee2d49fededdd39d771ec0141d4d64fccddf8d84192755e323" Jul 05 13:45:11.272777 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.272400 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.280203 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.280181 2569 scope.go:117] "RemoveContainer" containerID="d63f6dc734487ead525480da9ffa93d437178e5745bae2ec61364e1e9bb10462" Jul 05 13:45:11.287682 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.287663 2569 scope.go:117] "RemoveContainer" containerID="c151811ceb7d8664c1b89879696360be221df6c604c96e933e7159676c63e6b1" Jul 05 13:45:11.294057 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.294041 2569 scope.go:117] "RemoveContainer" containerID="cf4bc044c8256dcb3407debe1cd939cc94c2fd3c668ace66f8bdf33712d220b4" Jul 05 13:45:11.297983 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.297959 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:45:11.301654 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.301638 2569 scope.go:117] "RemoveContainer" containerID="353740a09da822960ce2d7aeebdedef8de1b18bd9ce41e479df6c4d9f2166215" Jul 05 13:45:11.302048 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.302023 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:45:11.307879 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.307858 2569 scope.go:117] "RemoveContainer" containerID="67d47b344e931959a61660f028617c47ce8c9c8a305fe1ef27fb64eb33c0e772" Jul 05 13:45:11.314555 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.314541 2569 scope.go:117] "RemoveContainer" containerID="c555be320bd1429726a41727db5c6e66f1673b83b9b2df7a412b893247ce10e2" Jul 05 13:45:11.333153 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333133 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:45:11.333478 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333461 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="init-config-reloader" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333482 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="init-config-reloader" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333496 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9f932b5-0dfc-47eb-80e2-b43594bf36e7" containerName="console" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333504 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9f932b5-0dfc-47eb-80e2-b43594bf36e7" containerName="console" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333522 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73359273-7ce0-4963-8590-b4e07880c348" containerName="registry" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333531 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="73359273-7ce0-4963-8590-b4e07880c348" containerName="registry" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333545 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-web" Jul 05 13:45:11.333564 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333554 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-web" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333571 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333578 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333590 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-thanos" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333599 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-thanos" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333609 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="prometheus" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333617 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="prometheus" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333630 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="thanos-sidecar" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333640 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="thanos-sidecar" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333651 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="config-reloader" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333662 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="config-reloader" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333727 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="73359273-7ce0-4963-8590-b4e07880c348" containerName="registry" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333740 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="prometheus" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333749 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9f932b5-0dfc-47eb-80e2-b43594bf36e7" containerName="console" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333757 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-thanos" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333769 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="config-reloader" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333778 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333785 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="thanos-sidecar" Jul 05 13:45:11.333893 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.333793 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" containerName="kube-rbac-proxy-web" Jul 05 13:45:11.339198 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.339155 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.342021 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.341998 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 05 13:45:11.342102 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342038 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 05 13:45:11.342102 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342052 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 05 13:45:11.342181 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342109 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 05 13:45:11.342397 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342379 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 05 13:45:11.342523 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342379 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 05 13:45:11.342523 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342437 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 05 13:45:11.342523 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342456 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 05 13:45:11.342523 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342491 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-111ohm7n1mmrs\"" Jul 05 13:45:11.342793 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342777 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 05 13:45:11.342866 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.342782 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 05 13:45:11.343500 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.343482 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 05 13:45:11.343603 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.343516 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 05 13:45:11.344267 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.344251 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 05 13:45:11.344496 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.344481 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-kqkrr\"" Jul 05 13:45:11.346640 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.346619 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 05 13:45:11.348300 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.348282 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 05 13:45:11.353633 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.353613 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:45:11.387076 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387049 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387175 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387175 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-config\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387175 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387175 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387167 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76001d73-0111-4c67-b04f-0f04e63491cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387229 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387270 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387318 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t976s\" (UniqueName: \"kubernetes.io/projected/76001d73-0111-4c67-b04f-0f04e63491cb-kube-api-access-t976s\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387353 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387409 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387403 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387458 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387543 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76001d73-0111-4c67-b04f-0f04e63491cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387584 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387630 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387676 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.387720 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.387699 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488238 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488215 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488359 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488242 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488359 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488359 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488293 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488359 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-config\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488393 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488410 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488429 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76001d73-0111-4c67-b04f-0f04e63491cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488449 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488489 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488518 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t976s\" (UniqueName: \"kubernetes.io/projected/76001d73-0111-4c67-b04f-0f04e63491cb-kube-api-access-t976s\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488575 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488639 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488655 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76001d73-0111-4c67-b04f-0f04e63491cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.488954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.488775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.489272 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.489012 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.489272 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.489048 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.489272 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.489131 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.489740 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.489716 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.489825 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.489780 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.490309 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.490284 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.490580 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.490556 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.492799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.492773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/76001d73-0111-4c67-b04f-0f04e63491cb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.493162 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.493117 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.493328 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.493305 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/76001d73-0111-4c67-b04f-0f04e63491cb-config-out\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.493509 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.493483 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.493948 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.493906 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/76001d73-0111-4c67-b04f-0f04e63491cb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.494585 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.493968 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-web-config\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.494585 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.494343 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.494585 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.494558 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.494826 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.494805 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.495158 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.495139 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-config\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.495847 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.495823 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.496261 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.496239 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/76001d73-0111-4c67-b04f-0f04e63491cb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.509235 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.509216 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t976s\" (UniqueName: \"kubernetes.io/projected/76001d73-0111-4c67-b04f-0f04e63491cb-kube-api-access-t976s\") pod \"prometheus-k8s-0\" (UID: \"76001d73-0111-4c67-b04f-0f04e63491cb\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.649091 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.649029 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:45:11.794994 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:11.794951 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 05 13:45:11.801518 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:45:11.801498 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76001d73_0111_4c67_b04f_0f04e63491cb.slice/crio-084b13cf38755c0e80bba119453923ee49530a8e77aff6f7afe815858412d150 WatchSource:0}: Error finding container 084b13cf38755c0e80bba119453923ee49530a8e77aff6f7afe815858412d150: Status 404 returned error can't find the container with id 084b13cf38755c0e80bba119453923ee49530a8e77aff6f7afe815858412d150 Jul 05 13:45:12.278603 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:12.278565 2569 generic.go:356] "Generic (PLEG): container finished" podID="76001d73-0111-4c67-b04f-0f04e63491cb" containerID="c0e4ae06fbe4ee141d05667f795bdb3b604b8f1bd76394a860d7b3b8c0191b7e" exitCode=0 Jul 05 13:45:12.278962 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:12.278611 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerDied","Data":"c0e4ae06fbe4ee141d05667f795bdb3b604b8f1bd76394a860d7b3b8c0191b7e"} Jul 05 13:45:12.278962 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:12.278675 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"084b13cf38755c0e80bba119453923ee49530a8e77aff6f7afe815858412d150"} Jul 05 13:45:12.799163 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:12.799075 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1bd76c4-f260-45b5-a232-df186b56bd71" path="/var/lib/kubelet/pods/e1bd76c4-f260-45b5-a232-df186b56bd71/volumes" Jul 05 13:45:13.286375 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.286331 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"36675443906a7e144c678774a5bf6ec6fa85169b29dee5f8c895c8b4607b3ce4"} Jul 05 13:45:13.286751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.286410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"786205be4d81b12457a715cfca99a28b5269df464d9eed2f1d14e1bbbf05a957"} Jul 05 13:45:13.286751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.286437 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"c2b37a51df380154bc5cbc5c6d5009996da48e9e75e3422f988f6a066cff0655"} Jul 05 13:45:13.286751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.286463 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"e675ec9727f53d65c059c726436aa9397fe25232e78aef8721d59cf35e39bde8"} Jul 05 13:45:13.286751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.286485 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"576dadd9af99ffd37e8c2ad970c3ee3001ef743eeacca7fd67a12f2810e03697"} Jul 05 13:45:13.286751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.286508 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"76001d73-0111-4c67-b04f-0f04e63491cb","Type":"ContainerStarted","Data":"421ff68bd2e9a1fde36324e5d0e55b47ab5239feb0861400f3b6e8da0681564c"} Jul 05 13:45:13.319659 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:13.319604 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.296566521 podStartE2EDuration="2.319590012s" totalImagesPullingTime="0s" totalInitContainerRuntime="23.023491ms" isStatefulPod=false podCreationTimestamp="2026-07-05 13:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:45:13.316580837 +0000 UTC m=+401.326050487" watchObservedRunningTime="2026-07-05 13:45:13.319590012 +0000 UTC m=+401.329059632" Jul 05 13:45:16.649313 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:45:16.649267 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:46:11.649360 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:11.649331 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:46:11.664786 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:11.664763 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:46:12.535248 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:12.535221 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 05 13:46:59.776303 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.776269 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh"] Jul 05 13:46:59.779305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.779290 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:46:59.784931 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.784905 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 05 13:46:59.785064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.784968 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 05 13:46:59.785064 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.785009 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 05 13:46:59.785183 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.784949 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 05 13:46:59.785183 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.784912 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 05 13:46:59.785380 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.785346 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zp774\"" Jul 05 13:46:59.797761 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.797740 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh"] Jul 05 13:46:59.925035 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.924998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ece36f5-e414-4ba6-9167-8a0529f6f372-manager-config\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:46:59.925377 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.925342 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ece36f5-e414-4ba6-9167-8a0529f6f372-metrics-cert\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:46:59.925560 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.925542 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgxjz\" (UniqueName: \"kubernetes.io/projected/4ece36f5-e414-4ba6-9167-8a0529f6f372-kube-api-access-wgxjz\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:46:59.925712 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:46:59.925697 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ece36f5-e414-4ba6-9167-8a0529f6f372-cert\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.026722 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.026634 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ece36f5-e414-4ba6-9167-8a0529f6f372-cert\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.026722 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.026712 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ece36f5-e414-4ba6-9167-8a0529f6f372-manager-config\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.026898 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.026738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ece36f5-e414-4ba6-9167-8a0529f6f372-metrics-cert\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.026898 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.026757 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgxjz\" (UniqueName: \"kubernetes.io/projected/4ece36f5-e414-4ba6-9167-8a0529f6f372-kube-api-access-wgxjz\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.027511 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.027484 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4ece36f5-e414-4ba6-9167-8a0529f6f372-manager-config\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.029068 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.029040 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4ece36f5-e414-4ba6-9167-8a0529f6f372-metrics-cert\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.029166 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.029085 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4ece36f5-e414-4ba6-9167-8a0529f6f372-cert\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.044908 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.044885 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgxjz\" (UniqueName: \"kubernetes.io/projected/4ece36f5-e414-4ba6-9167-8a0529f6f372-kube-api-access-wgxjz\") pod \"lws-controller-manager-bf64756c4-7rbzh\" (UID: \"4ece36f5-e414-4ba6-9167-8a0529f6f372\") " pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.089071 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.089050 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:00.248238 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.248215 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh"] Jul 05 13:47:00.249976 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:47:00.249947 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ece36f5_e414_4ba6_9167_8a0529f6f372.slice/crio-1592ab7d6c83397bc70d550361b6888bf4a86e6268a4118e534ad302a3cee263 WatchSource:0}: Error finding container 1592ab7d6c83397bc70d550361b6888bf4a86e6268a4118e534ad302a3cee263: Status 404 returned error can't find the container with id 1592ab7d6c83397bc70d550361b6888bf4a86e6268a4118e534ad302a3cee263 Jul 05 13:47:00.701297 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:00.701266 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" event={"ID":"4ece36f5-e414-4ba6-9167-8a0529f6f372","Type":"ContainerStarted","Data":"1592ab7d6c83397bc70d550361b6888bf4a86e6268a4118e534ad302a3cee263"} Jul 05 13:47:02.460907 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.460873 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf"] Jul 05 13:47:02.464770 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.464739 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.467505 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.467481 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 05 13:47:02.467645 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.467595 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 05 13:47:02.467716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.467690 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 05 13:47:02.468146 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.468123 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x926h\"" Jul 05 13:47:02.468254 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.468209 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 05 13:47:02.486623 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.486589 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf"] Jul 05 13:47:02.548980 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.548946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd2rx\" (UniqueName: \"kubernetes.io/projected/e152330b-409b-438e-acfd-b34426e9c030-kube-api-access-cd2rx\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.549147 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.549080 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e152330b-409b-438e-acfd-b34426e9c030-manifests\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.549147 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.549117 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e152330b-409b-438e-acfd-b34426e9c030-tmp\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.549147 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.549142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e152330b-409b-438e-acfd-b34426e9c030-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.549305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.549179 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e152330b-409b-438e-acfd-b34426e9c030-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.649729 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.649709 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e152330b-409b-438e-acfd-b34426e9c030-manifests\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.649847 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.649749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e152330b-409b-438e-acfd-b34426e9c030-tmp\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.649847 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.649775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e152330b-409b-438e-acfd-b34426e9c030-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.649847 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.649803 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e152330b-409b-438e-acfd-b34426e9c030-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.650002 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.649899 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd2rx\" (UniqueName: \"kubernetes.io/projected/e152330b-409b-438e-acfd-b34426e9c030-kube-api-access-cd2rx\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.650195 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.650128 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e152330b-409b-438e-acfd-b34426e9c030-manifests\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.652388 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.652328 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e152330b-409b-438e-acfd-b34426e9c030-tmp\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.652488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.652446 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e152330b-409b-438e-acfd-b34426e9c030-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.652743 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.652721 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e152330b-409b-438e-acfd-b34426e9c030-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.666442 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.666422 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd2rx\" (UniqueName: \"kubernetes.io/projected/e152330b-409b-438e-acfd-b34426e9c030-kube-api-access-cd2rx\") pod \"opendatahub-operator-controller-manager-5f79c67cfc-lpnrf\" (UID: \"e152330b-409b-438e-acfd-b34426e9c030\") " pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.778110 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.778071 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:02.938577 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:02.938551 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf"] Jul 05 13:47:02.940989 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:47:02.940962 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode152330b_409b_438e_acfd_b34426e9c030.slice/crio-98e6c7f66131e1886c280116309c4c6e419ec550957ce2d8beef820d072ac62f WatchSource:0}: Error finding container 98e6c7f66131e1886c280116309c4c6e419ec550957ce2d8beef820d072ac62f: Status 404 returned error can't find the container with id 98e6c7f66131e1886c280116309c4c6e419ec550957ce2d8beef820d072ac62f Jul 05 13:47:03.718747 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:03.718696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" event={"ID":"e152330b-409b-438e-acfd-b34426e9c030","Type":"ContainerStarted","Data":"98e6c7f66131e1886c280116309c4c6e419ec550957ce2d8beef820d072ac62f"} Jul 05 13:47:03.720828 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:03.720794 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" event={"ID":"4ece36f5-e414-4ba6-9167-8a0529f6f372","Type":"ContainerStarted","Data":"fba11c8c80ef7256d053af735d55dcf06fca629a600d4713fa07f0c94361fb1d"} Jul 05 13:47:03.721337 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:03.721167 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:03.755216 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:03.754808 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" podStartSLOduration=2.36426418 podStartE2EDuration="4.754790646s" totalImagesPullingTime="2.390526466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:47:03.753314311 +0000 UTC m=+511.762783953" watchObservedRunningTime="2026-07-05 13:47:03.754790646 +0000 UTC m=+511.764260267" Jul 05 13:47:06.734681 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:06.734649 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" event={"ID":"e152330b-409b-438e-acfd-b34426e9c030","Type":"ContainerStarted","Data":"28d79f127bff4fe8b58e209e4632178eeac6651dac629793170a547a643f8d49"} Jul 05 13:47:07.740433 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:07.740397 2569 generic.go:356] "Generic (PLEG): container finished" podID="e152330b-409b-438e-acfd-b34426e9c030" containerID="28d79f127bff4fe8b58e209e4632178eeac6651dac629793170a547a643f8d49" exitCode=0 Jul 05 13:47:07.740806 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:07.740454 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" event={"ID":"e152330b-409b-438e-acfd-b34426e9c030","Type":"ContainerDied","Data":"28d79f127bff4fe8b58e209e4632178eeac6651dac629793170a547a643f8d49"} Jul 05 13:47:08.746864 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:08.746827 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" event={"ID":"e152330b-409b-438e-acfd-b34426e9c030","Type":"ContainerStarted","Data":"2bb95ee83d013521fed946ad767c5228e3558f3b65363cb16382ba36981e469e"} Jul 05 13:47:08.747246 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:08.746916 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:08.771587 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:08.771534 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" podStartSLOduration=2.656010336 podStartE2EDuration="6.771518029s" totalImagesPullingTime="3.092503836s" totalInitContainerRuntime="1.023003857s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:47:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:47:08.769932167 +0000 UTC m=+516.779401797" watchObservedRunningTime="2026-07-05 13:47:08.771518029 +0000 UTC m=+516.780987652" Jul 05 13:47:14.727888 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:14.727847 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-bf64756c4-7rbzh" Jul 05 13:47:19.753068 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:19.753031 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f79c67cfc-lpnrf" Jul 05 13:47:28.575386 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.575290 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68d7598454-ttp9l"] Jul 05 13:47:28.578498 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.578483 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.581515 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.581491 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 05 13:47:28.581515 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.581508 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 05 13:47:28.581736 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.581611 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vdbz5\"" Jul 05 13:47:28.588962 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.588939 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68d7598454-ttp9l"] Jul 05 13:47:28.656826 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.656796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1064c7e-2d88-487b-b7ec-393a50a361ad-tmp\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.656965 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.656837 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqh6n\" (UniqueName: \"kubernetes.io/projected/c1064c7e-2d88-487b-b7ec-393a50a361ad-kube-api-access-bqh6n\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.656965 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.656875 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1064c7e-2d88-487b-b7ec-393a50a361ad-tls-certs\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.757883 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.757851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1064c7e-2d88-487b-b7ec-393a50a361ad-tls-certs\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.758056 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.757921 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1064c7e-2d88-487b-b7ec-393a50a361ad-tmp\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.758056 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.757951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqh6n\" (UniqueName: \"kubernetes.io/projected/c1064c7e-2d88-487b-b7ec-393a50a361ad-kube-api-access-bqh6n\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.760208 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.760176 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1064c7e-2d88-487b-b7ec-393a50a361ad-tmp\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.760418 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.760398 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1064c7e-2d88-487b-b7ec-393a50a361ad-tls-certs\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.767274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.767252 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqh6n\" (UniqueName: \"kubernetes.io/projected/c1064c7e-2d88-487b-b7ec-393a50a361ad-kube-api-access-bqh6n\") pod \"kube-auth-proxy-68d7598454-ttp9l\" (UID: \"c1064c7e-2d88-487b-b7ec-393a50a361ad\") " pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:28.888133 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:28.888064 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" Jul 05 13:47:29.021466 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:29.021286 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68d7598454-ttp9l"] Jul 05 13:47:29.024034 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:47:29.024002 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1064c7e_2d88_487b_b7ec_393a50a361ad.slice/crio-a22003f7386b14037a05084589e1626eb769014faa6cd8be895583f65edea1ad WatchSource:0}: Error finding container a22003f7386b14037a05084589e1626eb769014faa6cd8be895583f65edea1ad: Status 404 returned error can't find the container with id a22003f7386b14037a05084589e1626eb769014faa6cd8be895583f65edea1ad Jul 05 13:47:29.838176 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:29.838099 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" event={"ID":"c1064c7e-2d88-487b-b7ec-393a50a361ad","Type":"ContainerStarted","Data":"a22003f7386b14037a05084589e1626eb769014faa6cd8be895583f65edea1ad"} Jul 05 13:47:31.848557 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:31.848525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" event={"ID":"c1064c7e-2d88-487b-b7ec-393a50a361ad","Type":"ContainerStarted","Data":"5873547891cac87ad8043a7340af1c202735cfe1f0f18def4417755860c283be"} Jul 05 13:47:31.872478 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:31.872434 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68d7598454-ttp9l" podStartSLOduration=1.4630683979999999 podStartE2EDuration="3.872419454s" totalImagesPullingTime="2.409351056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:47:31.872016386 +0000 UTC m=+539.881486018" watchObservedRunningTime="2026-07-05 13:47:31.872419454 +0000 UTC m=+539.881889074" Jul 05 13:47:40.646026 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.645990 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s"] Jul 05 13:47:40.650684 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.650664 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.657261 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.657236 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 05 13:47:40.657380 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.657344 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x9qmv\"" Jul 05 13:47:40.682295 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.682270 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s"] Jul 05 13:47:40.758000 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.757972 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758133 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a657-778a-458b-baec-0adbd300445c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758133 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758035 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b81a657-778a-458b-baec-0adbd300445c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758133 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758133 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758348 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758348 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758210 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mknkz\" (UniqueName: \"kubernetes.io/projected/0b81a657-778a-458b-baec-0adbd300445c-kube-api-access-mknkz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758348 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758245 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.758348 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.758274 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b81a657-778a-458b-baec-0adbd300445c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.858687 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858653 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a657-778a-458b-baec-0adbd300445c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.858820 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b81a657-778a-458b-baec-0adbd300445c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.858820 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.858891 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858833 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.858932 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.858932 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mknkz\" (UniqueName: \"kubernetes.io/projected/0b81a657-778a-458b-baec-0adbd300445c-kube-api-access-mknkz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859007 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859007 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.858991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b81a657-778a-458b-baec-0adbd300445c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859106 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.859062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.859111 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859159 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.859141 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859312 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.859289 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859467 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.859447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.859549 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.859532 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b81a657-778a-458b-baec-0adbd300445c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.861252 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.861230 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b81a657-778a-458b-baec-0adbd300445c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.861478 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.861459 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b81a657-778a-458b-baec-0adbd300445c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.868829 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.868810 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b81a657-778a-458b-baec-0adbd300445c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.869119 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.869098 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mknkz\" (UniqueName: \"kubernetes.io/projected/0b81a657-778a-458b-baec-0adbd300445c-kube-api-access-mknkz\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678fpm6s\" (UID: \"0b81a657-778a-458b-baec-0adbd300445c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:40.960111 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:40.960087 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:41.103274 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:41.103241 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s"] Jul 05 13:47:41.107935 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:47:41.107909 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b81a657_778a_458b_baec_0adbd300445c.slice/crio-5a4254f305777250d66386207ccb1e25cb1d6c3d718d3704b41816ac11ab70dd WatchSource:0}: Error finding container 5a4254f305777250d66386207ccb1e25cb1d6c3d718d3704b41816ac11ab70dd: Status 404 returned error can't find the container with id 5a4254f305777250d66386207ccb1e25cb1d6c3d718d3704b41816ac11ab70dd Jul 05 13:47:41.894804 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:41.894758 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" event={"ID":"0b81a657-778a-458b-baec-0adbd300445c","Type":"ContainerStarted","Data":"5a4254f305777250d66386207ccb1e25cb1d6c3d718d3704b41816ac11ab70dd"} Jul 05 13:47:43.682906 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:43.682872 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:47:43.683162 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:43.682945 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:47:43.683162 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:43.682976 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:47:43.905722 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:43.905684 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" event={"ID":"0b81a657-778a-458b-baec-0adbd300445c","Type":"ContainerStarted","Data":"e16f92221306793cc10cb16bf6a7d367e9155bcc1985b4c5c4565d8674a2d69b"} Jul 05 13:47:43.936075 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:43.935961 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" podStartSLOduration=1.363576082 podStartE2EDuration="3.935942588s" totalImagesPullingTime="2.572366506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:47:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:47:43.934077394 +0000 UTC m=+551.943547015" watchObservedRunningTime="2026-07-05 13:47:43.935942588 +0000 UTC m=+551.945412210" Jul 05 13:47:43.960319 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:43.960294 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:44.964803 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:44.964777 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:45.913957 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:45.913928 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:47:45.914883 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:47:45.914862 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678fpm6s" Jul 05 13:49:00.639912 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.639880 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b5cdf44d-9dmng"] Jul 05 13:49:00.643149 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.643128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.646268 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.646244 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 05 13:49:00.646460 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.646446 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 05 13:49:00.647920 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.647889 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 05 13:49:00.648037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.647927 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-8z5r9\"" Jul 05 13:49:00.648037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.647897 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 05 13:49:00.648037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.647937 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 05 13:49:00.648037 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.647991 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 05 13:49:00.648214 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.648045 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 05 13:49:00.657204 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.657182 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 05 13:49:00.657310 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.657260 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b5cdf44d-9dmng"] Jul 05 13:49:00.772682 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g66v9\" (UniqueName: \"kubernetes.io/projected/d555297d-8b75-40e5-af88-41e4f578bb4b-kube-api-access-g66v9\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.772808 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-service-ca\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.772808 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-oauth-serving-cert\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.772808 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772734 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d555297d-8b75-40e5-af88-41e4f578bb4b-console-oauth-config\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.772974 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772852 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d555297d-8b75-40e5-af88-41e4f578bb4b-console-serving-cert\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.772974 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772884 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-trusted-ca-bundle\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.772974 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.772927 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-console-config\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873535 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873505 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d555297d-8b75-40e5-af88-41e4f578bb4b-console-serving-cert\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873667 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-trusted-ca-bundle\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873667 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873567 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-console-config\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873667 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g66v9\" (UniqueName: \"kubernetes.io/projected/d555297d-8b75-40e5-af88-41e4f578bb4b-kube-api-access-g66v9\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873667 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-service-ca\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873871 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873676 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-oauth-serving-cert\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.873871 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.873713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d555297d-8b75-40e5-af88-41e4f578bb4b-console-oauth-config\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.874342 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.874320 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-console-config\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.874554 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.874533 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-oauth-serving-cert\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.874624 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.874533 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-service-ca\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.874740 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.874717 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d555297d-8b75-40e5-af88-41e4f578bb4b-trusted-ca-bundle\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.876099 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.876071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d555297d-8b75-40e5-af88-41e4f578bb4b-console-oauth-config\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.876313 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.876293 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d555297d-8b75-40e5-af88-41e4f578bb4b-console-serving-cert\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.890934 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.890890 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g66v9\" (UniqueName: \"kubernetes.io/projected/d555297d-8b75-40e5-af88-41e4f578bb4b-kube-api-access-g66v9\") pod \"console-5b5cdf44d-9dmng\" (UID: \"d555297d-8b75-40e5-af88-41e4f578bb4b\") " pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:00.952786 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:00.952766 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:01.116828 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:01.116805 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b5cdf44d-9dmng"] Jul 05 13:49:01.118756 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:49:01.118730 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd555297d_8b75_40e5_af88_41e4f578bb4b.slice/crio-3574396b65319d07bd75d95e0a317389ab32ba3ede9ff8bcb1e50680ab514642 WatchSource:0}: Error finding container 3574396b65319d07bd75d95e0a317389ab32ba3ede9ff8bcb1e50680ab514642: Status 404 returned error can't find the container with id 3574396b65319d07bd75d95e0a317389ab32ba3ede9ff8bcb1e50680ab514642 Jul 05 13:49:01.231332 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:01.231293 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b5cdf44d-9dmng" event={"ID":"d555297d-8b75-40e5-af88-41e4f578bb4b","Type":"ContainerStarted","Data":"39a6a773aaa75e5990ba69fa83048149c8671b3e7ff2f9b97c8900bba020f7b0"} Jul 05 13:49:01.231495 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:01.231405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b5cdf44d-9dmng" event={"ID":"d555297d-8b75-40e5-af88-41e4f578bb4b","Type":"ContainerStarted","Data":"3574396b65319d07bd75d95e0a317389ab32ba3ede9ff8bcb1e50680ab514642"} Jul 05 13:49:01.260010 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:01.259946 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b5cdf44d-9dmng" podStartSLOduration=1.259929071 podStartE2EDuration="1.259929071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:49:01.259160683 +0000 UTC m=+629.268630303" watchObservedRunningTime="2026-07-05 13:49:01.259929071 +0000 UTC m=+629.269398673" Jul 05 13:49:10.953450 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:10.953419 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:10.953981 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:10.953475 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:10.958258 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:10.958236 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:11.287134 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:11.287054 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b5cdf44d-9dmng" Jul 05 13:49:20.428750 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.428717 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l"] Jul 05 13:49:20.431954 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.431934 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.434751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.434727 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 05 13:49:20.434751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.434738 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 05 13:49:20.435155 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.435132 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ljxll\"" Jul 05 13:49:20.436250 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.436234 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 05 13:49:20.436250 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.436243 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 05 13:49:20.442200 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.442182 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l"] Jul 05 13:49:20.449915 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.449895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.450011 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.449924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.450011 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.449943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drlfk\" (UniqueName: \"kubernetes.io/projected/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-kube-api-access-drlfk\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.550581 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.550555 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.550753 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.550593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.550753 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.550625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drlfk\" (UniqueName: \"kubernetes.io/projected/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-kube-api-access-drlfk\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.551261 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.551237 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.552967 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.552944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.563256 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.563231 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drlfk\" (UniqueName: \"kubernetes.io/projected/1d78f9b5-cf15-4df9-8ebe-28a1594a3b97-kube-api-access-drlfk\") pod \"kuadrant-console-plugin-66b5774985-hcl5l\" (UID: \"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.747342 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.747313 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" Jul 05 13:49:20.885780 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.885755 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l"] Jul 05 13:49:20.888458 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:49:20.888433 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d78f9b5_cf15_4df9_8ebe_28a1594a3b97.slice/crio-73dcfc6b4d6785aff1075bb3ffb9ad19bbbca53fb184a42c4a09fdd4b61d5316 WatchSource:0}: Error finding container 73dcfc6b4d6785aff1075bb3ffb9ad19bbbca53fb184a42c4a09fdd4b61d5316: Status 404 returned error can't find the container with id 73dcfc6b4d6785aff1075bb3ffb9ad19bbbca53fb184a42c4a09fdd4b61d5316 Jul 05 13:49:20.890303 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:20.890283 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 13:49:21.326871 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:21.326839 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" event={"ID":"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97","Type":"ContainerStarted","Data":"73dcfc6b4d6785aff1075bb3ffb9ad19bbbca53fb184a42c4a09fdd4b61d5316"} Jul 05 13:49:41.434091 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:41.434054 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" event={"ID":"1d78f9b5-cf15-4df9-8ebe-28a1594a3b97","Type":"ContainerStarted","Data":"0f421382dae455697253fae58c108f1b9ec055a2a33d2648fda45275a36eb2ff"} Jul 05 13:49:41.462261 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:49:41.462180 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-hcl5l" podStartSLOduration=1.150064552 podStartE2EDuration="21.462168259s" totalImagesPullingTime="20.312103707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:49:41.461452374 +0000 UTC m=+669.470921996" watchObservedRunningTime="2026-07-05 13:49:41.462168259 +0000 UTC m=+669.471637879" Jul 05 13:50:02.120939 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.120903 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz"] Jul 05 13:50:02.239612 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.239566 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz"] Jul 05 13:50:02.239612 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.239598 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.243514 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.243493 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kdv88\"" Jul 05 13:50:02.423776 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.423662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424071 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424071 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424230 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424111 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ng8n\" (UniqueName: \"kubernetes.io/projected/aa1dd6b7-4688-4c89-87bf-2c49e3535502-kube-api-access-7ng8n\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424230 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424216 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424310 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424310 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424416 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.424416 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.424384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525029 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.524996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525196 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525048 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525196 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ng8n\" (UniqueName: \"kubernetes.io/projected/aa1dd6b7-4688-4c89-87bf-2c49e3535502-kube-api-access-7ng8n\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525196 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525116 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525196 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525454 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525205 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525454 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525259 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525454 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525454 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525333 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525650 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525492 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525650 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525540 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.525879 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.525835 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.526545 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.526071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.526545 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.526098 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.527798 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.527769 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.528768 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.528741 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.536355 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.536322 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1dd6b7-4688-4c89-87bf-2c49e3535502-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.536895 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.536874 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ng8n\" (UniqueName: \"kubernetes.io/projected/aa1dd6b7-4688-4c89-87bf-2c49e3535502-kube-api-access-7ng8n\") pod \"maas-default-gateway-openshift-default-5896d4dd49-pnxpz\" (UID: \"aa1dd6b7-4688-4c89-87bf-2c49e3535502\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.550335 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.550310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:02.729242 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.729212 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz"] Jul 05 13:50:02.731503 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:50:02.731478 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1dd6b7_4688_4c89_87bf_2c49e3535502.slice/crio-01e667ed5c832ff27d202cf7a60baec42e3f0821c93d24cfc64634481c8143f6 WatchSource:0}: Error finding container 01e667ed5c832ff27d202cf7a60baec42e3f0821c93d24cfc64634481c8143f6: Status 404 returned error can't find the container with id 01e667ed5c832ff27d202cf7a60baec42e3f0821c93d24cfc64634481c8143f6 Jul 05 13:50:02.733708 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.733666 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:50:02.733799 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.733755 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:50:02.733839 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:02.733798 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:50:03.536446 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:03.536410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" event={"ID":"aa1dd6b7-4688-4c89-87bf-2c49e3535502","Type":"ContainerStarted","Data":"4abb22018c51957d689c5abadc94114aa83c3dd3db825d372bece26025c86671"} Jul 05 13:50:03.536896 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:03.536467 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" event={"ID":"aa1dd6b7-4688-4c89-87bf-2c49e3535502","Type":"ContainerStarted","Data":"01e667ed5c832ff27d202cf7a60baec42e3f0821c93d24cfc64634481c8143f6"} Jul 05 13:50:03.551042 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:03.551019 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:03.556224 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:03.556189 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:03.564530 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:03.564490 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" podStartSLOduration=1.564477241 podStartE2EDuration="1.564477241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:50:03.562060355 +0000 UTC m=+691.571530001" watchObservedRunningTime="2026-07-05 13:50:03.564477241 +0000 UTC m=+691.573946920" Jul 05 13:50:04.540527 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:04.540496 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:04.541353 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:04.541335 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:50:05.462099 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.462069 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-6pbsn"] Jul 05 13:50:05.533860 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.533827 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6pbsn"] Jul 05 13:50:05.534009 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.533917 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:05.537305 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.537283 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hqqhs\"" Jul 05 13:50:05.551490 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.551467 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b576q\" (UniqueName: \"kubernetes.io/projected/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a-kube-api-access-b576q\") pod \"authorino-58cd955675-6pbsn\" (UID: \"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a\") " pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:05.652302 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.652263 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b576q\" (UniqueName: \"kubernetes.io/projected/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a-kube-api-access-b576q\") pod \"authorino-58cd955675-6pbsn\" (UID: \"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a\") " pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:05.662088 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.662063 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b576q\" (UniqueName: \"kubernetes.io/projected/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a-kube-api-access-b576q\") pod \"authorino-58cd955675-6pbsn\" (UID: \"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a\") " pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:05.844414 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:05.844342 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:06.019636 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:06.019609 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-6pbsn"] Jul 05 13:50:06.022069 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:50:06.022038 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1112c6d1_aaf8_4fc9_a44e_4a30be0cfc7a.slice/crio-5d7caa38e0938c64ed38d2013d119f4123ccac8b4d7ad4d635f6f51b17a9db73 WatchSource:0}: Error finding container 5d7caa38e0938c64ed38d2013d119f4123ccac8b4d7ad4d635f6f51b17a9db73: Status 404 returned error can't find the container with id 5d7caa38e0938c64ed38d2013d119f4123ccac8b4d7ad4d635f6f51b17a9db73 Jul 05 13:50:06.550230 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:06.550198 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6pbsn" event={"ID":"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a","Type":"ContainerStarted","Data":"5d7caa38e0938c64ed38d2013d119f4123ccac8b4d7ad4d635f6f51b17a9db73"} Jul 05 13:50:10.575161 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:10.575127 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6pbsn" event={"ID":"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a","Type":"ContainerStarted","Data":"7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89"} Jul 05 13:50:10.602522 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:10.602476 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-6pbsn" podStartSLOduration=1.647103355 podStartE2EDuration="5.602461086s" totalImagesPullingTime="3.955357731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:50:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:50:10.601671529 +0000 UTC m=+698.611141151" watchObservedRunningTime="2026-07-05 13:50:10.602461086 +0000 UTC m=+698.611930705" Jul 05 13:50:10.895045 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:10.894966 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6pbsn"] Jul 05 13:50:12.583492 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:12.583455 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-6pbsn" podUID="1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" containerName="authorino" containerID="cri-o://7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89" gracePeriod=30 Jul 05 13:50:12.826983 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:12.826959 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:12.921672 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:12.921603 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b576q\" (UniqueName: \"kubernetes.io/projected/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a-kube-api-access-b576q\") pod \"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a\" (UID: \"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a\") " Jul 05 13:50:12.923780 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:12.923750 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a-kube-api-access-b576q" (OuterVolumeSpecName: "kube-api-access-b576q") pod "1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" (UID: "1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a"). InnerVolumeSpecName "kube-api-access-b576q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:50:13.022248 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.022219 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b576q\" (UniqueName: \"kubernetes.io/projected/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a-kube-api-access-b576q\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:50:13.588855 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.588817 2569 generic.go:356] "Generic (PLEG): container finished" podID="1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" containerID="7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89" exitCode=0 Jul 05 13:50:13.588855 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.588860 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-6pbsn" Jul 05 13:50:13.589326 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.588880 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6pbsn" event={"ID":"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a","Type":"ContainerDied","Data":"7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89"} Jul 05 13:50:13.589326 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.588918 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-6pbsn" event={"ID":"1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a","Type":"ContainerDied","Data":"5d7caa38e0938c64ed38d2013d119f4123ccac8b4d7ad4d635f6f51b17a9db73"} Jul 05 13:50:13.589326 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.588940 2569 scope.go:117] "RemoveContainer" containerID="7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89" Jul 05 13:50:13.598204 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.598185 2569 scope.go:117] "RemoveContainer" containerID="7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89" Jul 05 13:50:13.598553 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:50:13.598535 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89\": container with ID starting with 7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89 not found: ID does not exist" containerID="7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89" Jul 05 13:50:13.598625 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.598558 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89"} err="failed to get container status \"7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89\": rpc error: code = NotFound desc = could not find container \"7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89\": container with ID starting with 7a75ccafd050dfa30adeb80eaa9ed20e976bdec33b8ef55e73c4b65ea0546d89 not found: ID does not exist" Jul 05 13:50:13.618768 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.618746 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-6pbsn"] Jul 05 13:50:13.624020 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:13.623999 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-6pbsn"] Jul 05 13:50:14.798259 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:14.798225 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" path="/var/lib/kubelet/pods/1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a/volumes" Jul 05 13:50:37.910436 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.910379 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqcbh"] Jul 05 13:50:37.910891 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.910735 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" containerName="authorino" Jul 05 13:50:37.910891 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.910749 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" containerName="authorino" Jul 05 13:50:37.910891 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.910816 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1112c6d1-aaf8-4fc9-a44e-4a30be0cfc7a" containerName="authorino" Jul 05 13:50:37.915075 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.915055 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:37.918150 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.918131 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hqqhs\"" Jul 05 13:50:37.922566 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:37.922455 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqcbh"] Jul 05 13:50:38.033337 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.033298 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72fr\" (UniqueName: \"kubernetes.io/projected/ceba4063-f0ae-4699-b820-a9d6ea955283-kube-api-access-c72fr\") pod \"authorino-5d4b67c446-bqcbh\" (UID: \"ceba4063-f0ae-4699-b820-a9d6ea955283\") " pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:38.134280 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.134246 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72fr\" (UniqueName: \"kubernetes.io/projected/ceba4063-f0ae-4699-b820-a9d6ea955283-kube-api-access-c72fr\") pod \"authorino-5d4b67c446-bqcbh\" (UID: \"ceba4063-f0ae-4699-b820-a9d6ea955283\") " pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:38.144034 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.144001 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72fr\" (UniqueName: \"kubernetes.io/projected/ceba4063-f0ae-4699-b820-a9d6ea955283-kube-api-access-c72fr\") pod \"authorino-5d4b67c446-bqcbh\" (UID: \"ceba4063-f0ae-4699-b820-a9d6ea955283\") " pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:38.144479 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.144456 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqcbh"] Jul 05 13:50:38.144734 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.144718 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:38.276175 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.276144 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqcbh"] Jul 05 13:50:38.279793 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:50:38.279764 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podceba4063_f0ae_4699_b820_a9d6ea955283.slice/crio-8684578f409042912f1daa514aceca98bee08b15194910b14e7b0763e05a3ae5 WatchSource:0}: Error finding container 8684578f409042912f1daa514aceca98bee08b15194910b14e7b0763e05a3ae5: Status 404 returned error can't find the container with id 8684578f409042912f1daa514aceca98bee08b15194910b14e7b0763e05a3ae5 Jul 05 13:50:38.306048 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.306023 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cd469b7d5-89c5w"] Jul 05 13:50:38.310344 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.310327 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.313300 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.313282 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 05 13:50:38.319264 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.319244 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd469b7d5-89c5w"] Jul 05 13:50:38.437391 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.437337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6m4q\" (UniqueName: \"kubernetes.io/projected/e7dbb869-f5e0-41f6-8486-031649bcd099-kube-api-access-n6m4q\") pod \"authorino-cd469b7d5-89c5w\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.437391 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.437395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dbb869-f5e0-41f6-8486-031649bcd099-tls-cert\") pod \"authorino-cd469b7d5-89c5w\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.538594 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.538565 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6m4q\" (UniqueName: \"kubernetes.io/projected/e7dbb869-f5e0-41f6-8486-031649bcd099-kube-api-access-n6m4q\") pod \"authorino-cd469b7d5-89c5w\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.538747 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.538601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dbb869-f5e0-41f6-8486-031649bcd099-tls-cert\") pod \"authorino-cd469b7d5-89c5w\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.540996 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.540973 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dbb869-f5e0-41f6-8486-031649bcd099-tls-cert\") pod \"authorino-cd469b7d5-89c5w\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.548658 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.548636 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6m4q\" (UniqueName: \"kubernetes.io/projected/e7dbb869-f5e0-41f6-8486-031649bcd099-kube-api-access-n6m4q\") pod \"authorino-cd469b7d5-89c5w\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.620201 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.620176 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:50:38.705096 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.705056 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" event={"ID":"ceba4063-f0ae-4699-b820-a9d6ea955283","Type":"ContainerStarted","Data":"c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58"} Jul 05 13:50:38.705276 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.705108 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" event={"ID":"ceba4063-f0ae-4699-b820-a9d6ea955283","Type":"ContainerStarted","Data":"8684578f409042912f1daa514aceca98bee08b15194910b14e7b0763e05a3ae5"} Jul 05 13:50:38.705276 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.705134 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" podUID="ceba4063-f0ae-4699-b820-a9d6ea955283" containerName="authorino" containerID="cri-o://c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58" gracePeriod=30 Jul 05 13:50:38.723283 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.723239 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" podStartSLOduration=1.439760425 podStartE2EDuration="1.723225197s" totalImagesPullingTime="283.464772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:50:38.721680386 +0000 UTC m=+726.731150007" watchObservedRunningTime="2026-07-05 13:50:38.723225197 +0000 UTC m=+726.732694818" Jul 05 13:50:38.764868 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.764841 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cd469b7d5-89c5w"] Jul 05 13:50:38.768220 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:50:38.768191 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7dbb869_f5e0_41f6_8486_031649bcd099.slice/crio-1198cf26066566a282cbedce938619442fd1420ffe0a90d7e1430a64938f742e WatchSource:0}: Error finding container 1198cf26066566a282cbedce938619442fd1420ffe0a90d7e1430a64938f742e: Status 404 returned error can't find the container with id 1198cf26066566a282cbedce938619442fd1420ffe0a90d7e1430a64938f742e Jul 05 13:50:38.941182 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:38.941157 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:39.042285 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.042253 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c72fr\" (UniqueName: \"kubernetes.io/projected/ceba4063-f0ae-4699-b820-a9d6ea955283-kube-api-access-c72fr\") pod \"ceba4063-f0ae-4699-b820-a9d6ea955283\" (UID: \"ceba4063-f0ae-4699-b820-a9d6ea955283\") " Jul 05 13:50:39.044238 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.044213 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceba4063-f0ae-4699-b820-a9d6ea955283-kube-api-access-c72fr" (OuterVolumeSpecName: "kube-api-access-c72fr") pod "ceba4063-f0ae-4699-b820-a9d6ea955283" (UID: "ceba4063-f0ae-4699-b820-a9d6ea955283"). InnerVolumeSpecName "kube-api-access-c72fr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:50:39.143614 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.143538 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72fr\" (UniqueName: \"kubernetes.io/projected/ceba4063-f0ae-4699-b820-a9d6ea955283-kube-api-access-c72fr\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:50:39.711468 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.711430 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd469b7d5-89c5w" event={"ID":"e7dbb869-f5e0-41f6-8486-031649bcd099","Type":"ContainerStarted","Data":"ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2"} Jul 05 13:50:39.711657 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.711480 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd469b7d5-89c5w" event={"ID":"e7dbb869-f5e0-41f6-8486-031649bcd099","Type":"ContainerStarted","Data":"1198cf26066566a282cbedce938619442fd1420ffe0a90d7e1430a64938f742e"} Jul 05 13:50:39.712593 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.712568 2569 generic.go:356] "Generic (PLEG): container finished" podID="ceba4063-f0ae-4699-b820-a9d6ea955283" containerID="c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58" exitCode=0 Jul 05 13:50:39.712679 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.712604 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" Jul 05 13:50:39.712679 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.712648 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" event={"ID":"ceba4063-f0ae-4699-b820-a9d6ea955283","Type":"ContainerDied","Data":"c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58"} Jul 05 13:50:39.712751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.712684 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-bqcbh" event={"ID":"ceba4063-f0ae-4699-b820-a9d6ea955283","Type":"ContainerDied","Data":"8684578f409042912f1daa514aceca98bee08b15194910b14e7b0763e05a3ae5"} Jul 05 13:50:39.712751 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.712703 2569 scope.go:117] "RemoveContainer" containerID="c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58" Jul 05 13:50:39.721083 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.721057 2569 scope.go:117] "RemoveContainer" containerID="c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58" Jul 05 13:50:39.721342 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:50:39.721322 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58\": container with ID starting with c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58 not found: ID does not exist" containerID="c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58" Jul 05 13:50:39.721421 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.721351 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58"} err="failed to get container status \"c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58\": rpc error: code = NotFound desc = could not find container \"c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58\": container with ID starting with c20234fb59aa0f448e460ff2663098ade9a0db6f61e8b10ff8f428690b227c58 not found: ID does not exist" Jul 05 13:50:39.731020 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.730981 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-cd469b7d5-89c5w" podStartSLOduration=1.3685952430000001 podStartE2EDuration="1.730967927s" totalImagesPullingTime="362.372684ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:50:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:50:39.729801937 +0000 UTC m=+727.739271556" watchObservedRunningTime="2026-07-05 13:50:39.730967927 +0000 UTC m=+727.740437544" Jul 05 13:50:39.748570 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.748545 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqcbh"] Jul 05 13:50:39.753787 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:39.753726 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-bqcbh"] Jul 05 13:50:40.799309 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:50:40.799274 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ceba4063-f0ae-4699-b820-a9d6ea955283" path="/var/lib/kubelet/pods/ceba4063-f0ae-4699-b820-a9d6ea955283/volumes" Jul 05 13:51:19.456678 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.456647 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-z5jmr"] Jul 05 13:51:19.457127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.457004 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ceba4063-f0ae-4699-b820-a9d6ea955283" containerName="authorino" Jul 05 13:51:19.457127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.457019 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ceba4063-f0ae-4699-b820-a9d6ea955283" containerName="authorino" Jul 05 13:51:19.457127 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.457103 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ceba4063-f0ae-4699-b820-a9d6ea955283" containerName="authorino" Jul 05 13:51:19.460160 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.460143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.462840 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.462818 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 05 13:51:19.471052 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.471030 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-z5jmr"] Jul 05 13:51:19.601329 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.601292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lj2\" (UniqueName: \"kubernetes.io/projected/733e1f4d-73ae-409a-8408-22589ea86b20-kube-api-access-q9lj2\") pod \"payload-pre-processing-7964457696-z5jmr\" (UID: \"733e1f4d-73ae-409a-8408-22589ea86b20\") " pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.601495 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.601357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733e1f4d-73ae-409a-8408-22589ea86b20-plugins-config-volume\") pod \"payload-pre-processing-7964457696-z5jmr\" (UID: \"733e1f4d-73ae-409a-8408-22589ea86b20\") " pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.702604 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.702574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733e1f4d-73ae-409a-8408-22589ea86b20-plugins-config-volume\") pod \"payload-pre-processing-7964457696-z5jmr\" (UID: \"733e1f4d-73ae-409a-8408-22589ea86b20\") " pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.702791 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.702629 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lj2\" (UniqueName: \"kubernetes.io/projected/733e1f4d-73ae-409a-8408-22589ea86b20-kube-api-access-q9lj2\") pod \"payload-pre-processing-7964457696-z5jmr\" (UID: \"733e1f4d-73ae-409a-8408-22589ea86b20\") " pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.703333 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.703308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/733e1f4d-73ae-409a-8408-22589ea86b20-plugins-config-volume\") pod \"payload-pre-processing-7964457696-z5jmr\" (UID: \"733e1f4d-73ae-409a-8408-22589ea86b20\") " pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.711856 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.711803 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lj2\" (UniqueName: \"kubernetes.io/projected/733e1f4d-73ae-409a-8408-22589ea86b20-kube-api-access-q9lj2\") pod \"payload-pre-processing-7964457696-z5jmr\" (UID: \"733e1f4d-73ae-409a-8408-22589ea86b20\") " pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.771324 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.771298 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:19.898949 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:19.898926 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-z5jmr"] Jul 05 13:51:19.901109 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:51:19.901084 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733e1f4d_73ae_409a_8408_22589ea86b20.slice/crio-07c0bbb01901ce3759224d5c53929470a790e7af17a29206b69f41be564726eb WatchSource:0}: Error finding container 07c0bbb01901ce3759224d5c53929470a790e7af17a29206b69f41be564726eb: Status 404 returned error can't find the container with id 07c0bbb01901ce3759224d5c53929470a790e7af17a29206b69f41be564726eb Jul 05 13:51:20.890792 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:20.890756 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" event={"ID":"733e1f4d-73ae-409a-8408-22589ea86b20","Type":"ContainerStarted","Data":"07c0bbb01901ce3759224d5c53929470a790e7af17a29206b69f41be564726eb"} Jul 05 13:51:26.926104 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:26.926068 2569 generic.go:356] "Generic (PLEG): container finished" podID="aa1dd6b7-4688-4c89-87bf-2c49e3535502" containerID="4abb22018c51957d689c5abadc94114aa83c3dd3db825d372bece26025c86671" exitCode=0 Jul 05 13:51:26.926460 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:26.926141 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" event={"ID":"aa1dd6b7-4688-4c89-87bf-2c49e3535502","Type":"ContainerDied","Data":"4abb22018c51957d689c5abadc94114aa83c3dd3db825d372bece26025c86671"} Jul 05 13:51:26.926522 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:26.926508 2569 scope.go:117] "RemoveContainer" containerID="4abb22018c51957d689c5abadc94114aa83c3dd3db825d372bece26025c86671" Jul 05 13:51:26.927550 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:26.927517 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:51:26.927618 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:26.927587 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:51:26.927654 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:26.927620 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 13:51:27.937485 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:27.937441 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" event={"ID":"aa1dd6b7-4688-4c89-87bf-2c49e3535502","Type":"ContainerStarted","Data":"8d60c05579a46020c4fa2ea58efc5c0ef20069b9c866fc6a1d7d44e4f3cacc0e"} Jul 05 13:51:28.551206 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:28.551169 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:51:28.551451 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:28.551428 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pnxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 05 13:51:28.551539 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:28.551486 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" podUID="aa1dd6b7-4688-4c89-87bf-2c49e3535502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 05 13:51:29.551099 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:29.551060 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pnxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 05 13:51:29.551488 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:29.551117 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" podUID="aa1dd6b7-4688-4c89-87bf-2c49e3535502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 05 13:51:30.228617 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:51:30.228565 2569 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/odh-ai-gateway-payload-processing:dc31b949d8c5ea8610b5c10ef53e2a199e00f0e0: reading manifest sha256:44c1927a333c20287b6b9ee8b89baa296119fbae73089872a87365a9785461c3 in quay.io/opendatahub/odh-ai-gateway-payload-processing: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" image="quay.io/opendatahub/odh-ai-gateway-payload-processing:dc31b949d8c5ea8610b5c10ef53e2a199e00f0e0" Jul 05 13:51:30.229010 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:51:30.228975 2569 kuberuntime_manager.go:1459] "Unhandled Error" err="container payload-pre-processing start failed in pod payload-pre-processing-7964457696-z5jmr_openshift-ingress(733e1f4d-73ae-409a-8408-22589ea86b20): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/odh-ai-gateway-payload-processing:dc31b949d8c5ea8610b5c10ef53e2a199e00f0e0: reading manifest sha256:44c1927a333c20287b6b9ee8b89baa296119fbae73089872a87365a9785461c3 in quay.io/opendatahub/odh-ai-gateway-payload-processing: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image" logger="UnhandledError" Jul 05 13:51:30.229443 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:51:30.229410 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"payload-pre-processing\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/odh-ai-gateway-payload-processing:dc31b949d8c5ea8610b5c10ef53e2a199e00f0e0: reading manifest sha256:44c1927a333c20287b6b9ee8b89baa296119fbae73089872a87365a9785461c3 in quay.io/opendatahub/odh-ai-gateway-payload-processing: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" podUID="733e1f4d-73ae-409a-8408-22589ea86b20" Jul 05 13:51:30.551231 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:30.551197 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pnxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 05 13:51:30.551647 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:30.551250 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" podUID="aa1dd6b7-4688-4c89-87bf-2c49e3535502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 05 13:51:30.954503 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:51:30.954460 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"payload-pre-processing\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/opendatahub/odh-ai-gateway-payload-processing:dc31b949d8c5ea8610b5c10ef53e2a199e00f0e0\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: determining manifest MIME type for docker://quay.io/opendatahub/odh-ai-gateway-payload-processing:dc31b949d8c5ea8610b5c10ef53e2a199e00f0e0: reading manifest sha256:44c1927a333c20287b6b9ee8b89baa296119fbae73089872a87365a9785461c3 in quay.io/opendatahub/odh-ai-gateway-payload-processing: received unexpected HTTP status: 504 Gateway Time-out; artifact err: provided artifact is a container image\"" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" podUID="733e1f4d-73ae-409a-8408-22589ea86b20" Jul 05 13:51:31.551672 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:31.551633 2569 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-pnxpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 05 13:51:31.552082 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:31.551689 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" podUID="aa1dd6b7-4688-4c89-87bf-2c49e3535502" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 05 13:51:32.555105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:32.555076 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:51:32.555494 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:32.555322 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:51:32.555935 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:32.555919 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-pnxpz" Jul 05 13:51:49.038842 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:49.038805 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" event={"ID":"733e1f4d-73ae-409a-8408-22589ea86b20","Type":"ContainerStarted","Data":"e873acfa61d70e3a43f3d676cdc17b1697b858c906c87e5cf529852393915547"} Jul 05 13:51:49.039252 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:49.039028 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:51:49.055638 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:49.055598 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" podStartSLOduration=1.900644381 podStartE2EDuration="30.055582957s" totalImagesPullingTime="28.154938576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:51:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-05 13:51:49.055099317 +0000 UTC m=+797.064568938" watchObservedRunningTime="2026-07-05 13:51:49.055582957 +0000 UTC m=+797.065052577" Jul 05 13:51:57.092500 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.092420 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t"] Jul 05 13:51:57.095985 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.095967 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.100271 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.100248 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tj5dq\"" Jul 05 13:51:57.100429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.100248 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 05 13:51:57.100429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.100289 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 05 13:51:57.100429 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.100305 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 05 13:51:57.104932 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.104903 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t"] Jul 05 13:51:57.236546 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.236512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3820e1f-f601-4880-8133-beb9b244cd1d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.236706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.236551 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.236706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.236578 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7sr9\" (UniqueName: \"kubernetes.io/projected/d3820e1f-f601-4880-8133-beb9b244cd1d-kube-api-access-d7sr9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.236706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.236662 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.236706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.236687 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.236706 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.236704 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.337582 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.337548 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3820e1f-f601-4880-8133-beb9b244cd1d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.337773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.337589 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.337773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.337614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7sr9\" (UniqueName: \"kubernetes.io/projected/d3820e1f-f601-4880-8133-beb9b244cd1d-kube-api-access-d7sr9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.337773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.337658 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.337773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.337739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.337773 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.337771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.338052 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.338010 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.338113 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.338081 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.338157 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.338119 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.339839 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.339819 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d3820e1f-f601-4880-8133-beb9b244cd1d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.340115 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.340097 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3820e1f-f601-4880-8133-beb9b244cd1d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.345623 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.345566 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7sr9\" (UniqueName: \"kubernetes.io/projected/d3820e1f-f601-4880-8133-beb9b244cd1d-kube-api-access-d7sr9\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t\" (UID: \"d3820e1f-f601-4880-8133-beb9b244cd1d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.407253 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.407228 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:51:57.539716 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:57.539693 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t"] Jul 05 13:51:57.541898 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:51:57.541854 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3820e1f_f601_4880_8133_beb9b244cd1d.slice/crio-a092365f1301691635f246828a6e4829d0d1b1fe06807d0bfeae92dfc3ae3dbf WatchSource:0}: Error finding container a092365f1301691635f246828a6e4829d0d1b1fe06807d0bfeae92dfc3ae3dbf: Status 404 returned error can't find the container with id a092365f1301691635f246828a6e4829d0d1b1fe06807d0bfeae92dfc3ae3dbf Jul 05 13:51:58.091477 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:58.091435 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" event={"ID":"d3820e1f-f601-4880-8133-beb9b244cd1d","Type":"ContainerStarted","Data":"a092365f1301691635f246828a6e4829d0d1b1fe06807d0bfeae92dfc3ae3dbf"} Jul 05 13:51:59.098051 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:59.098018 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" event={"ID":"d3820e1f-f601-4880-8133-beb9b244cd1d","Type":"ContainerStarted","Data":"715e6b685a5320755a53b8d07ce92eb9310b6684e51a9b670f49825d10942150"} Jul 05 13:51:59.126674 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:51:59.126628 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" podStartSLOduration=1.150133011 podStartE2EDuration="2.126613526s" totalImagesPullingTime="976.480515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:51:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:51:59.124583232 +0000 UTC m=+807.134052871" watchObservedRunningTime="2026-07-05 13:51:59.126613526 +0000 UTC m=+807.136083145" Jul 05 13:52:00.046031 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:00.046002 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-z5jmr" Jul 05 13:52:07.407811 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:07.407774 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:52:07.408230 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:07.407836 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:52:07.420260 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:07.420232 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:52:08.149791 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:08.149765 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t" Jul 05 13:52:17.493289 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.493251 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw"] Jul 05 13:52:17.496870 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.496852 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.499838 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.499815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 05 13:52:17.505673 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.505654 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw"] Jul 05 13:52:17.607008 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.606974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.607168 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.607055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.607168 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.607105 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.607265 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.607168 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.607265 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.607224 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.607343 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.607284 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scvf\" (UniqueName: \"kubernetes.io/projected/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-kube-api-access-2scvf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.708611 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.708576 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.708797 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.708630 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scvf\" (UniqueName: \"kubernetes.io/projected/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-kube-api-access-2scvf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.708797 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.708652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.708797 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.708709 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.708797 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.708751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.708797 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.708787 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.709086 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.709056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.709141 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.709105 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.709202 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.709143 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.710992 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.710975 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.711402 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.711382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.717057 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.717034 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scvf\" (UniqueName: \"kubernetes.io/projected/96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6-kube-api-access-2scvf\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw\" (UID: \"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.807105 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.807009 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:17.935076 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:17.935051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw"] Jul 05 13:52:17.937070 ip-10-0-130-182 kubenswrapper[2569]: W0705 13:52:17.937028 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96e584a0_fc7e_4e91_ab47_7f04a1ffe6b6.slice/crio-24b28db4813d2835c1aa638c30515f980507c0cd5801f9c509ad83989e0c1ee0 WatchSource:0}: Error finding container 24b28db4813d2835c1aa638c30515f980507c0cd5801f9c509ad83989e0c1ee0: Status 404 returned error can't find the container with id 24b28db4813d2835c1aa638c30515f980507c0cd5801f9c509ad83989e0c1ee0 Jul 05 13:52:18.188486 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:18.188456 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" event={"ID":"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6","Type":"ContainerStarted","Data":"6e1800182f018d89b79b7f40178f3bb6e9de71ae5a7baca601a05d244e02673a"} Jul 05 13:52:18.188595 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:18.188502 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" event={"ID":"96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6","Type":"ContainerStarted","Data":"24b28db4813d2835c1aa638c30515f980507c0cd5801f9c509ad83989e0c1ee0"} Jul 05 13:52:18.207132 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:18.207090 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" podStartSLOduration=1.033571902 podStartE2EDuration="1.207075012s" totalImagesPullingTime="173.50311ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 13:52:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 13:52:18.205614471 +0000 UTC m=+826.215084092" watchObservedRunningTime="2026-07-05 13:52:18.207075012 +0000 UTC m=+826.216544687" Jul 05 13:52:27.807590 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:27.807549 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:27.808004 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:27.807603 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:27.820069 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:27.820046 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:52:28.248641 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:52:28.248615 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw" Jul 05 13:53:11.805071 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:11.805025 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cd469b7d5-89c5w"] Jul 05 13:53:11.805663 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:11.805226 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-cd469b7d5-89c5w" podUID="e7dbb869-f5e0-41f6-8486-031649bcd099" containerName="authorino" containerID="cri-o://ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2" gracePeriod=30 Jul 05 13:53:12.050431 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.050410 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:53:12.174480 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.174406 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dbb869-f5e0-41f6-8486-031649bcd099-tls-cert\") pod \"e7dbb869-f5e0-41f6-8486-031649bcd099\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " Jul 05 13:53:12.174611 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.174507 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n6m4q\" (UniqueName: \"kubernetes.io/projected/e7dbb869-f5e0-41f6-8486-031649bcd099-kube-api-access-n6m4q\") pod \"e7dbb869-f5e0-41f6-8486-031649bcd099\" (UID: \"e7dbb869-f5e0-41f6-8486-031649bcd099\") " Jul 05 13:53:12.176340 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.176302 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7dbb869-f5e0-41f6-8486-031649bcd099-kube-api-access-n6m4q" (OuterVolumeSpecName: "kube-api-access-n6m4q") pod "e7dbb869-f5e0-41f6-8486-031649bcd099" (UID: "e7dbb869-f5e0-41f6-8486-031649bcd099"). InnerVolumeSpecName "kube-api-access-n6m4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 13:53:12.184355 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.184329 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7dbb869-f5e0-41f6-8486-031649bcd099-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e7dbb869-f5e0-41f6-8486-031649bcd099" (UID: "e7dbb869-f5e0-41f6-8486-031649bcd099"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 13:53:12.275542 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.275516 2569 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e7dbb869-f5e0-41f6-8486-031649bcd099-tls-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:53:12.275542 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.275539 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6m4q\" (UniqueName: \"kubernetes.io/projected/e7dbb869-f5e0-41f6-8486-031649bcd099-kube-api-access-n6m4q\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 13:53:12.443581 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.443514 2569 generic.go:356] "Generic (PLEG): container finished" podID="e7dbb869-f5e0-41f6-8486-031649bcd099" containerID="ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2" exitCode=0 Jul 05 13:53:12.443581 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.443557 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cd469b7d5-89c5w" Jul 05 13:53:12.443714 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.443590 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd469b7d5-89c5w" event={"ID":"e7dbb869-f5e0-41f6-8486-031649bcd099","Type":"ContainerDied","Data":"ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2"} Jul 05 13:53:12.443714 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.443634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cd469b7d5-89c5w" event={"ID":"e7dbb869-f5e0-41f6-8486-031649bcd099","Type":"ContainerDied","Data":"1198cf26066566a282cbedce938619442fd1420ffe0a90d7e1430a64938f742e"} Jul 05 13:53:12.443714 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.443651 2569 scope.go:117] "RemoveContainer" containerID="ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2" Jul 05 13:53:12.452933 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.452918 2569 scope.go:117] "RemoveContainer" containerID="ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2" Jul 05 13:53:12.453174 ip-10-0-130-182 kubenswrapper[2569]: E0705 13:53:12.453156 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2\": container with ID starting with ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2 not found: ID does not exist" containerID="ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2" Jul 05 13:53:12.453220 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.453182 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2"} err="failed to get container status \"ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2\": rpc error: code = NotFound desc = could not find container \"ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2\": container with ID starting with ba1a6bd1dbb5f256241be1a7739238862cacc26c050876c7b775b1db08955aa2 not found: ID does not exist" Jul 05 13:53:12.468933 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.468908 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-cd469b7d5-89c5w"] Jul 05 13:53:12.473888 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.473869 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-cd469b7d5-89c5w"] Jul 05 13:53:12.799221 ip-10-0-130-182 kubenswrapper[2569]: I0705 13:53:12.799188 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7dbb869-f5e0-41f6-8486-031649bcd099" path="/var/lib/kubelet/pods/e7dbb869-f5e0-41f6-8486-031649bcd099/volumes" Jul 05 14:06:24.517237 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.517204 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt"] Jul 05 14:06:24.519505 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.517747 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7dbb869-f5e0-41f6-8486-031649bcd099" containerName="authorino" Jul 05 14:06:24.519505 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.517764 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7dbb869-f5e0-41f6-8486-031649bcd099" containerName="authorino" Jul 05 14:06:24.519505 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.517836 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7dbb869-f5e0-41f6-8486-031649bcd099" containerName="authorino" Jul 05 14:06:24.520823 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.520804 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.524118 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.524093 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ca40252-openshift-default-dockercfg-klgps\"" Jul 05 14:06:24.546581 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.546552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt"] Jul 05 14:06:24.598042 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-certs\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598135 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598050 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-credential-socket\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598135 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-envoy\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598209 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-socket\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598209 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-data\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598276 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598223 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-podinfo\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598276 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598244 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-token\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598276 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598265 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74171743-7f0c-4352-b0ab-9241e3a3ec19-istiod-ca-cert\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.598397 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.598299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55z4m\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-kube-api-access-55z4m\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.698881 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.698853 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-podinfo\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.698891 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-token\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.698909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74171743-7f0c-4352-b0ab-9241e3a3ec19-istiod-ca-cert\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.698928 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55z4m\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-kube-api-access-55z4m\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.698963 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-certs\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.698986 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-credential-socket\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699010 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-envoy\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-socket\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699439 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-data\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699525 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-data\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699637 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699547 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-certs\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699637 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699594 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-credential-socket\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699753 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699638 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-socket\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.699753 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.699668 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74171743-7f0c-4352-b0ab-9241e3a3ec19-istiod-ca-cert\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.701447 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.701418 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-envoy\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.701575 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.701560 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-podinfo\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.708084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.708060 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-token\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.708166 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.708129 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55z4m\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-kube-api-access-55z4m\") pod \"e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.831550 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.831493 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:24.962563 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.962540 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt"] Jul 05 14:06:24.964543 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:06:24.964514 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74171743_7f0c_4352_b0ab_9241e3a3ec19.slice/crio-f6bc313dcb22c4985ea2970a22dbfaca846fffb591596cb21b2fb517d5cb86e3 WatchSource:0}: Error finding container f6bc313dcb22c4985ea2970a22dbfaca846fffb591596cb21b2fb517d5cb86e3: Status 404 returned error can't find the container with id f6bc313dcb22c4985ea2970a22dbfaca846fffb591596cb21b2fb517d5cb86e3 Jul 05 14:06:24.966714 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.966681 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:24.966786 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.966751 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:24.966825 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:24.966797 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:25.046392 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:25.046337 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" event={"ID":"74171743-7f0c-4352-b0ab-9241e3a3ec19","Type":"ContainerStarted","Data":"db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8"} Jul 05 14:06:25.046510 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:25.046411 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" event={"ID":"74171743-7f0c-4352-b0ab-9241e3a3ec19","Type":"ContainerStarted","Data":"f6bc313dcb22c4985ea2970a22dbfaca846fffb591596cb21b2fb517d5cb86e3"} Jul 05 14:06:25.070640 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:25.070585 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" podStartSLOduration=1.070566109 podStartE2EDuration="1.070566109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:06:25.069123093 +0000 UTC m=+1673.078592771" watchObservedRunningTime="2026-07-05 14:06:25.070566109 +0000 UTC m=+1673.080035729" Jul 05 14:06:25.832603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:25.832570 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:26.837749 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:26.837723 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:27.056767 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:27.056738 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:27.057658 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:27.057642 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:31.125781 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:31.125736 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt"] Jul 05 14:06:31.126284 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:31.125963 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" podUID="74171743-7f0c-4352-b0ab-9241e3a3ec19" containerName="istio-proxy" containerID="cri-o://db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8" gracePeriod=30 Jul 05 14:06:36.581860 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.581830 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:36.711665 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711640 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-certs\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.711813 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711678 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-data\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.711813 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711728 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74171743-7f0c-4352-b0ab-9241e3a3ec19-istiod-ca-cert\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.711813 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711760 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-credential-socket\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.711813 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711795 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-envoy\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.712033 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711825 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-podinfo\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.712033 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711857 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-55z4m\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-kube-api-access-55z4m\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.712033 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711892 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-socket\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.712033 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711908 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:36.712033 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.711934 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-token\") pod \"74171743-7f0c-4352-b0ab-9241e3a3ec19\" (UID: \"74171743-7f0c-4352-b0ab-9241e3a3ec19\") " Jul 05 14:06:36.712312 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712094 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:36.712312 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712164 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74171743-7f0c-4352-b0ab-9241e3a3ec19-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:06:36.712445 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712303 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:36.712555 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712532 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.712603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712561 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.712603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712576 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74171743-7f0c-4352-b0ab-9241e3a3ec19-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.712603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712589 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.712730 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.712630 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-data" (OuterVolumeSpecName: "istio-data") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:36.714087 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.714062 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-token" (OuterVolumeSpecName: "istio-token") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:06:36.714199 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.714140 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:36.714481 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.714458 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-kube-api-access-55z4m" (OuterVolumeSpecName: "kube-api-access-55z4m") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "kube-api-access-55z4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:06:36.714481 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.714470 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74171743-7f0c-4352-b0ab-9241e3a3ec19" (UID: "74171743-7f0c-4352-b0ab-9241e3a3ec19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:06:36.812997 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.812974 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.812997 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.812997 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.813129 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.813006 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55z4m\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-kube-api-access-55z4m\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.813129 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.813016 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:36.813129 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:36.813025 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74171743-7f0c-4352-b0ab-9241e3a3ec19-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:37.104691 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.104611 2569 generic.go:356] "Generic (PLEG): container finished" podID="74171743-7f0c-4352-b0ab-9241e3a3ec19" containerID="db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8" exitCode=0 Jul 05 14:06:37.104691 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.104651 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" Jul 05 14:06:37.104691 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.104665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" event={"ID":"74171743-7f0c-4352-b0ab-9241e3a3ec19","Type":"ContainerDied","Data":"db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8"} Jul 05 14:06:37.104956 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.104718 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt" event={"ID":"74171743-7f0c-4352-b0ab-9241e3a3ec19","Type":"ContainerDied","Data":"f6bc313dcb22c4985ea2970a22dbfaca846fffb591596cb21b2fb517d5cb86e3"} Jul 05 14:06:37.104956 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.104746 2569 scope.go:117] "RemoveContainer" containerID="db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8" Jul 05 14:06:37.113403 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.113384 2569 scope.go:117] "RemoveContainer" containerID="db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8" Jul 05 14:06:37.113645 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:06:37.113624 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8\": container with ID starting with db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8 not found: ID does not exist" containerID="db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8" Jul 05 14:06:37.113732 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.113650 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8"} err="failed to get container status \"db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8\": rpc error: code = NotFound desc = could not find container \"db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8\": container with ID starting with db294ccbe41dd478feb51b62a352470feac60a0d95a73626a8a7f8214533cae8 not found: ID does not exist" Jul 05 14:06:37.124611 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.124588 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt"] Jul 05 14:06:37.131625 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.131607 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ca40252-openshift-default-6fb6df8f7-x76rt"] Jul 05 14:06:37.696999 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.696950 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx"] Jul 05 14:06:37.697560 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.697540 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74171743-7f0c-4352-b0ab-9241e3a3ec19" containerName="istio-proxy" Jul 05 14:06:37.697613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.697564 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74171743-7f0c-4352-b0ab-9241e3a3ec19" containerName="istio-proxy" Jul 05 14:06:37.697662 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.697651 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74171743-7f0c-4352-b0ab-9241e3a3ec19" containerName="istio-proxy" Jul 05 14:06:37.706343 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.706320 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.711025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.710996 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b8ce2e85-openshift-default-dockercfg-hvhwf\"" Jul 05 14:06:37.725034 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.725008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx"] Jul 05 14:06:37.822306 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822275 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-envoy\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822482 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822321 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5pst\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-kube-api-access-g5pst\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822482 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-certs\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822482 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822433 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-podinfo\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822482 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822463 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-socket\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822650 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822517 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-data\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822650 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-credential-socket\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822650 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d05f9c-89b5-4423-9d9b-e44069b909de-istiod-ca-cert\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.822650 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.822616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-token\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.923646 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-envoy\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.923798 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5pst\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-kube-api-access-g5pst\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.923798 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923685 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-certs\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.923798 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923724 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-podinfo\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.923798 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923756 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-socket\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.923798 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-data\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923812 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-credential-socket\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923847 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d05f9c-89b5-4423-9d9b-e44069b909de-istiod-ca-cert\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.923871 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-token\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.924203 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-socket\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.924225 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-data\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.924361 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-certs\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924470 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.924436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-credential-socket\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.924842 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.924768 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d05f9c-89b5-4423-9d9b-e44069b909de-istiod-ca-cert\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.926006 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.925988 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-envoy\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.926062 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.926050 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-podinfo\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.933929 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.933889 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-token\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:37.934468 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:37.934447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5pst\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-kube-api-access-g5pst\") pod \"e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:38.018498 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:38.018470 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:38.159802 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:38.159768 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx"] Jul 05 14:06:38.162939 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:06:38.162913 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3d05f9c_89b5_4423_9d9b_e44069b909de.slice/crio-2bc1e6fdfbf34e03a4036628103d49c903c588ba7b0e3e7530e45f554868fbce WatchSource:0}: Error finding container 2bc1e6fdfbf34e03a4036628103d49c903c588ba7b0e3e7530e45f554868fbce: Status 404 returned error can't find the container with id 2bc1e6fdfbf34e03a4036628103d49c903c588ba7b0e3e7530e45f554868fbce Jul 05 14:06:38.165242 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:38.165195 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:38.165311 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:38.165292 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:38.165352 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:38.165339 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:38.800070 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:38.800035 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74171743-7f0c-4352-b0ab-9241e3a3ec19" path="/var/lib/kubelet/pods/74171743-7f0c-4352-b0ab-9241e3a3ec19/volumes" Jul 05 14:06:39.119220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:39.119137 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" event={"ID":"e3d05f9c-89b5-4423-9d9b-e44069b909de","Type":"ContainerStarted","Data":"7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a"} Jul 05 14:06:39.119220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:39.119185 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" event={"ID":"e3d05f9c-89b5-4423-9d9b-e44069b909de","Type":"ContainerStarted","Data":"2bc1e6fdfbf34e03a4036628103d49c903c588ba7b0e3e7530e45f554868fbce"} Jul 05 14:06:39.165121 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:39.165059 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" podStartSLOduration=2.165040109 podStartE2EDuration="2.165040109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:06:39.162017641 +0000 UTC m=+1687.171487275" watchObservedRunningTime="2026-07-05 14:06:39.165040109 +0000 UTC m=+1687.174509945" Jul 05 14:06:40.019671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:40.019641 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:40.025126 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:40.025100 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:40.123886 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:40.123848 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:40.124803 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:40.124784 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:50.474404 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:50.474358 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx"] Jul 05 14:06:50.474856 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:50.474553 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" podUID="e3d05f9c-89b5-4423-9d9b-e44069b909de" containerName="istio-proxy" containerID="cri-o://7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a" gracePeriod=30 Jul 05 14:06:55.126644 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.126428 2569 patch_prober.go:28] interesting pod/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 05 14:06:55.126644 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.126502 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" podUID="e3d05f9c-89b5-4423-9d9b-e44069b909de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 14:06:55.723870 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.723842 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:55.875750 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875669 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-podinfo\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.875750 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875730 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-certs\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.875973 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875785 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g5pst\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-kube-api-access-g5pst\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.875973 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875808 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-data\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.875973 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875827 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-token\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.875973 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875857 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-socket\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.875973 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875945 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-envoy\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.876220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875963 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:55.876220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.875983 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-credential-socket\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.876220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876049 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d05f9c-89b5-4423-9d9b-e44069b909de-istiod-ca-cert\") pod \"e3d05f9c-89b5-4423-9d9b-e44069b909de\" (UID: \"e3d05f9c-89b5-4423-9d9b-e44069b909de\") " Jul 05 14:06:55.876220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876072 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:55.876476 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876238 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:55.876476 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876264 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:55.876476 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876473 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.876613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876493 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.876613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876507 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.876613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876522 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.876613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.876516 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3d05f9c-89b5-4423-9d9b-e44069b909de-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:06:55.878095 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.878047 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:06:55.878095 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.878059 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-kube-api-access-g5pst" (OuterVolumeSpecName: "kube-api-access-g5pst") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "kube-api-access-g5pst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:06:55.878268 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.878146 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:06:55.878268 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.878245 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3d05f9c-89b5-4423-9d9b-e44069b909de" (UID: "e3d05f9c-89b5-4423-9d9b-e44069b909de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:06:55.977555 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.977529 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.977555 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.977550 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3d05f9c-89b5-4423-9d9b-e44069b909de-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.977555 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.977559 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.977739 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.977578 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5pst\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-kube-api-access-g5pst\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:55.977739 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:55.977589 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3d05f9c-89b5-4423-9d9b-e44069b909de-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:06:56.197080 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.197057 2569 generic.go:356] "Generic (PLEG): container finished" podID="e3d05f9c-89b5-4423-9d9b-e44069b909de" containerID="7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a" exitCode=0 Jul 05 14:06:56.197477 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.197097 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" Jul 05 14:06:56.197477 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.197143 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" event={"ID":"e3d05f9c-89b5-4423-9d9b-e44069b909de","Type":"ContainerDied","Data":"7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a"} Jul 05 14:06:56.197477 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.197197 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx" event={"ID":"e3d05f9c-89b5-4423-9d9b-e44069b909de","Type":"ContainerDied","Data":"2bc1e6fdfbf34e03a4036628103d49c903c588ba7b0e3e7530e45f554868fbce"} Jul 05 14:06:56.197477 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.197219 2569 scope.go:117] "RemoveContainer" containerID="7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a" Jul 05 14:06:56.206086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.206070 2569 scope.go:117] "RemoveContainer" containerID="7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a" Jul 05 14:06:56.206384 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:06:56.206347 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a\": container with ID starting with 7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a not found: ID does not exist" containerID="7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a" Jul 05 14:06:56.206444 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.206405 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a"} err="failed to get container status \"7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a\": rpc error: code = NotFound desc = could not find container \"7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a\": container with ID starting with 7e3e91b7b25ca939c38169a0213685f5871cfbad5ae5f380a2bf11a94eef433a not found: ID does not exist" Jul 05 14:06:56.235104 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.235081 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx"] Jul 05 14:06:56.242258 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.242240 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b8ce2e85-openshift-default-77876c6fc5-mj2jx"] Jul 05 14:06:56.798835 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.798806 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3d05f9c-89b5-4423-9d9b-e44069b909de" path="/var/lib/kubelet/pods/e3d05f9c-89b5-4423-9d9b-e44069b909de/volumes" Jul 05 14:06:56.988248 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.988211 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725"] Jul 05 14:06:56.988592 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.988579 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3d05f9c-89b5-4423-9d9b-e44069b909de" containerName="istio-proxy" Jul 05 14:06:56.988647 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.988594 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3d05f9c-89b5-4423-9d9b-e44069b909de" containerName="istio-proxy" Jul 05 14:06:56.988688 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.988666 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3d05f9c-89b5-4423-9d9b-e44069b909de" containerName="istio-proxy" Jul 05 14:06:56.991421 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.991402 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:56.994271 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:56.994246 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-cdb84f0d-openshift-default-dockercfg-84mfn\"" Jul 05 14:06:57.004610 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.004591 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725"] Jul 05 14:06:57.088002 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.087941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81144a05-d215-4166-9fc5-c8180ce10be4-istio-podinfo\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088002 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.087970 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81144a05-d215-4166-9fc5-c8180ce10be4-istiod-ca-cert\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088156 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-socket\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088156 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-credential-socket\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088156 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088109 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-istio-token\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088270 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-envoy\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088270 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088198 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbxcw\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-kube-api-access-jbxcw\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088270 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-certs\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.088390 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.088315 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-data\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189191 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189157 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-data\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189191 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81144a05-d215-4166-9fc5-c8180ce10be4-istio-podinfo\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189417 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189219 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81144a05-d215-4166-9fc5-c8180ce10be4-istiod-ca-cert\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189417 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-socket\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189417 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189319 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-credential-socket\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189417 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189341 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-istio-token\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189417 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189412 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-envoy\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189676 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189437 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbxcw\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-kube-api-access-jbxcw\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189676 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189467 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-certs\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189676 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189645 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-data\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189850 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189723 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-credential-socket\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.189850 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.189847 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-socket\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.190040 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.190018 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-certs\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.190323 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.190306 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81144a05-d215-4166-9fc5-c8180ce10be4-istiod-ca-cert\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.191704 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.191684 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-envoy\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.191922 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.191903 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81144a05-d215-4166-9fc5-c8180ce10be4-istio-podinfo\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.198430 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.198409 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbxcw\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-kube-api-access-jbxcw\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.198808 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.198473 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-istio-token\") pod \"e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.301757 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.301736 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:57.437688 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.437664 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725"] Jul 05 14:06:57.439545 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:06:57.439510 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81144a05_d215_4166_9fc5_c8180ce10be4.slice/crio-b62225a6b3cfccfbfe6483eb3445ba52fa74d67c4403c10184ad96981e6fb007 WatchSource:0}: Error finding container b62225a6b3cfccfbfe6483eb3445ba52fa74d67c4403c10184ad96981e6fb007: Status 404 returned error can't find the container with id b62225a6b3cfccfbfe6483eb3445ba52fa74d67c4403c10184ad96981e6fb007 Jul 05 14:06:57.441794 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.441761 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:57.441872 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.441863 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:57.441913 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:57.441895 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:06:58.208966 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:58.208931 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" event={"ID":"81144a05-d215-4166-9fc5-c8180ce10be4","Type":"ContainerStarted","Data":"63f98e27d21f781af802e782fb5035af6c63a5b67415e7f90391bb80697de0e7"} Jul 05 14:06:58.209325 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:58.208981 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" event={"ID":"81144a05-d215-4166-9fc5-c8180ce10be4","Type":"ContainerStarted","Data":"b62225a6b3cfccfbfe6483eb3445ba52fa74d67c4403c10184ad96981e6fb007"} Jul 05 14:06:58.240451 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:58.240395 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" podStartSLOduration=2.240376675 podStartE2EDuration="2.240376675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:06:58.238654083 +0000 UTC m=+1706.248123744" watchObservedRunningTime="2026-07-05 14:06:58.240376675 +0000 UTC m=+1706.249846292" Jul 05 14:06:58.301871 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:58.301845 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:58.306500 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:58.306478 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:59.214024 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:59.213989 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:06:59.214980 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:06:59.214958 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:07:02.971111 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:02.971069 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725"] Jul 05 14:07:02.971617 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:02.971336 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" podUID="81144a05-d215-4166-9fc5-c8180ce10be4" containerName="istio-proxy" containerID="cri-o://63f98e27d21f781af802e782fb5035af6c63a5b67415e7f90391bb80697de0e7" gracePeriod=30 Jul 05 14:07:08.276407 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.276352 2569 generic.go:356] "Generic (PLEG): container finished" podID="81144a05-d215-4166-9fc5-c8180ce10be4" containerID="63f98e27d21f781af802e782fb5035af6c63a5b67415e7f90391bb80697de0e7" exitCode=0 Jul 05 14:07:08.276866 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.276409 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" event={"ID":"81144a05-d215-4166-9fc5-c8180ce10be4","Type":"ContainerDied","Data":"63f98e27d21f781af802e782fb5035af6c63a5b67415e7f90391bb80697de0e7"} Jul 05 14:07:08.320045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.320026 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:07:08.397044 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.396990 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-envoy\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397044 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397023 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-certs\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397222 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397045 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81144a05-d215-4166-9fc5-c8180ce10be4-istiod-ca-cert\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397222 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397085 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-data\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397222 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397160 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-istio-token\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397222 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397193 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-socket\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397222 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397216 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-credential-socket\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397515 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397254 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81144a05-d215-4166-9fc5-c8180ce10be4-istio-podinfo\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397515 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397287 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:07:08.397515 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397301 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbxcw\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-kube-api-access-jbxcw\") pod \"81144a05-d215-4166-9fc5-c8180ce10be4\" (UID: \"81144a05-d215-4166-9fc5-c8180ce10be4\") " Jul 05 14:07:08.397672 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397522 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81144a05-d215-4166-9fc5-c8180ce10be4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:07:08.397672 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397528 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:07:08.397797 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397775 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:07:08.397852 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397838 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.397908 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397853 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.397908 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397862 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81144a05-d215-4166-9fc5-c8180ce10be4-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.398014 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.397991 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-data" (OuterVolumeSpecName: "istio-data") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:07:08.399330 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.399306 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81144a05-d215-4166-9fc5-c8180ce10be4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:07:08.399452 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.399394 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-istio-token" (OuterVolumeSpecName: "istio-token") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:07:08.399452 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.399433 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:07:08.399565 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.399527 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-kube-api-access-jbxcw" (OuterVolumeSpecName: "kube-api-access-jbxcw") pod "81144a05-d215-4166-9fc5-c8180ce10be4" (UID: "81144a05-d215-4166-9fc5-c8180ce10be4"). InnerVolumeSpecName "kube-api-access-jbxcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:07:08.498281 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.498256 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.498281 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.498281 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.498453 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.498295 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81144a05-d215-4166-9fc5-c8180ce10be4-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.498453 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.498310 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbxcw\" (UniqueName: \"kubernetes.io/projected/81144a05-d215-4166-9fc5-c8180ce10be4-kube-api-access-jbxcw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.498453 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.498323 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:08.498453 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:08.498337 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81144a05-d215-4166-9fc5-c8180ce10be4-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:07:09.287633 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:09.287598 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" Jul 05 14:07:09.287633 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:09.287612 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725" event={"ID":"81144a05-d215-4166-9fc5-c8180ce10be4","Type":"ContainerDied","Data":"b62225a6b3cfccfbfe6483eb3445ba52fa74d67c4403c10184ad96981e6fb007"} Jul 05 14:07:09.288190 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:09.287709 2569 scope.go:117] "RemoveContainer" containerID="63f98e27d21f781af802e782fb5035af6c63a5b67415e7f90391bb80697de0e7" Jul 05 14:07:09.310344 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:09.310322 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725"] Jul 05 14:07:09.316944 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:09.316919 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-cdb84f0d-openshift-default-c95f9bf87-zg725"] Jul 05 14:07:10.799851 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:07:10.799815 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81144a05-d215-4166-9fc5-c8180ce10be4" path="/var/lib/kubelet/pods/81144a05-d215-4166-9fc5-c8180ce10be4/volumes" Jul 05 14:08:46.284908 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.284876 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr"] Jul 05 14:08:46.285400 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.285230 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81144a05-d215-4166-9fc5-c8180ce10be4" containerName="istio-proxy" Jul 05 14:08:46.285400 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.285240 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="81144a05-d215-4166-9fc5-c8180ce10be4" containerName="istio-proxy" Jul 05 14:08:46.285400 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.285312 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="81144a05-d215-4166-9fc5-c8180ce10be4" containerName="istio-proxy" Jul 05 14:08:46.287835 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.287819 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.290778 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.290757 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-28ccab57-openshift-default-dockercfg-rk5k8\"" Jul 05 14:08:46.310875 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.310850 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr"] Jul 05 14:08:46.367779 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367755 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-envoy\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.367938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367796 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-socket\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.367938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367819 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-podinfo\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.367938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367871 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-credential-socket\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.367938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e94815-cbed-47cf-ad06-6a4c5783d571-istiod-ca-cert\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.367938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367915 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-data\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.368259 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-token\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.368259 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.367998 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frdjw\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-kube-api-access-frdjw\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.368259 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.368016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-certs\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469203 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469156 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-envoy\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469203 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469216 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-socket\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469236 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-podinfo\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469258 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-credential-socket\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469275 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e94815-cbed-47cf-ad06-6a4c5783d571-istiod-ca-cert\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-data\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-token\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frdjw\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-kube-api-access-frdjw\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469391 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-certs\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469852 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469732 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-data\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469852 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469802 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-socket\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469967 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469898 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-certs\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.469967 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.469957 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-credential-socket\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.470131 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.470110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e94815-cbed-47cf-ad06-6a4c5783d571-istiod-ca-cert\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.471520 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.471504 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-envoy\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.471853 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.471833 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-podinfo\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.477897 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.477874 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-token\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.478265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.478245 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frdjw\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-kube-api-access-frdjw\") pod \"e2e-mt-28ccab57-openshift-default-986b88495-x9wcr\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.598094 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.598031 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:46.733069 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.732868 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr"] Jul 05 14:08:46.735298 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:08:46.735266 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e94815_cbed_47cf_ad06_6a4c5783d571.slice/crio-cfdaa06c67686b4abc3dcfdf589c61e71e8191b9e00ccc40fc75df2ecdf5a0b3 WatchSource:0}: Error finding container cfdaa06c67686b4abc3dcfdf589c61e71e8191b9e00ccc40fc75df2ecdf5a0b3: Status 404 returned error can't find the container with id cfdaa06c67686b4abc3dcfdf589c61e71e8191b9e00ccc40fc75df2ecdf5a0b3 Jul 05 14:08:46.737558 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.737520 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:08:46.737671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.737590 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:08:46.737671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.737621 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:08:46.742944 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:46.742913 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" event={"ID":"34e94815-cbed-47cf-ad06-6a4c5783d571","Type":"ContainerStarted","Data":"cfdaa06c67686b4abc3dcfdf589c61e71e8191b9e00ccc40fc75df2ecdf5a0b3"} Jul 05 14:08:47.755382 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:47.755326 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" event={"ID":"34e94815-cbed-47cf-ad06-6a4c5783d571","Type":"ContainerStarted","Data":"2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6"} Jul 05 14:08:47.778575 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:47.778532 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" podStartSLOduration=1.778517433 podStartE2EDuration="1.778517433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:08:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:08:47.777473908 +0000 UTC m=+1815.786943553" watchObservedRunningTime="2026-07-05 14:08:47.778517433 +0000 UTC m=+1815.787987113" Jul 05 14:08:48.599160 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:48.599125 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:48.603602 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:48.603577 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:48.760649 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:48.760615 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:08:48.761577 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:08:48.761554 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:02.825542 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:02.825499 2569 generic.go:356] "Generic (PLEG): container finished" podID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerID="2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6" exitCode=0 Jul 05 14:09:02.825980 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:02.825572 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" event={"ID":"34e94815-cbed-47cf-ad06-6a4c5783d571","Type":"ContainerDied","Data":"2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6"} Jul 05 14:09:02.825980 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:02.825962 2569 scope.go:117] "RemoveContainer" containerID="2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6" Jul 05 14:09:02.826948 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:02.826921 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:02.827003 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:02.826991 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:02.827040 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:02.827025 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:03.760652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:03.760623 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:03.839410 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:03.839344 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" event={"ID":"34e94815-cbed-47cf-ad06-6a4c5783d571","Type":"ContainerStarted","Data":"bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e"} Jul 05 14:09:04.599273 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:04.599225 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:04.600532 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:04.600506 2569 patch_prober.go:28] interesting pod/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 05 14:09:04.600645 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:04.600570 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 05 14:09:05.599424 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:05.599357 2569 patch_prober.go:28] interesting pod/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 05 14:09:05.599847 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:05.599450 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 05 14:09:06.598929 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:06.598890 2569 patch_prober.go:28] interesting pod/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 05 14:09:06.599123 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:06.598949 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 05 14:09:07.598944 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:07.598907 2569 patch_prober.go:28] interesting pod/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 05 14:09:07.599336 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:07.598987 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 05 14:09:08.602792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:08.602761 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:08.603183 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:08.603117 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:08.603770 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:08.603753 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:08.706809 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:08.706772 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr"] Jul 05 14:09:08.866020 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:08.865930 2569 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" secret="" err="secret \"e2e-mt-28ccab57-openshift-default-dockercfg-rk5k8\" not found" Jul 05 14:09:09.870854 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:09.870811 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" containerID="cri-o://bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e" gracePeriod=30 Jul 05 14:09:15.129178 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.129118 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:15.141612 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141586 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-credential-socket\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141757 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141650 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-data\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141757 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-certs\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141757 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141695 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-envoy\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141757 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141754 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frdjw\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-kube-api-access-frdjw\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141961 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141805 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-podinfo\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141961 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141847 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-socket\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141961 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141878 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e94815-cbed-47cf-ad06-6a4c5783d571-istiod-ca-cert\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.141961 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141902 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:09:15.141961 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141909 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-token\") pod \"34e94815-cbed-47cf-ad06-6a4c5783d571\" (UID: \"34e94815-cbed-47cf-ad06-6a4c5783d571\") " Jul 05 14:09:15.142220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.141982 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:09:15.142275 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.142221 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:09:15.142328 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.142304 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.142404 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.142327 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.142467 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.142439 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e94815-cbed-47cf-ad06-6a4c5783d571-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:09:15.142525 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.142507 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-data" (OuterVolumeSpecName: "istio-data") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:09:15.144066 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.144035 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:09:15.144182 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.144153 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-token" (OuterVolumeSpecName: "istio-token") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:09:15.144342 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.144320 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-kube-api-access-frdjw" (OuterVolumeSpecName: "kube-api-access-frdjw") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "kube-api-access-frdjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:09:15.144342 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.144326 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "34e94815-cbed-47cf-ad06-6a4c5783d571" (UID: "34e94815-cbed-47cf-ad06-6a4c5783d571"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:09:15.243251 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243217 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.243251 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243246 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.243251 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243255 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.243522 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243267 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frdjw\" (UniqueName: \"kubernetes.io/projected/34e94815-cbed-47cf-ad06-6a4c5783d571-kube-api-access-frdjw\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.243522 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243277 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e94815-cbed-47cf-ad06-6a4c5783d571-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.243522 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243285 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e94815-cbed-47cf-ad06-6a4c5783d571-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.243522 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.243294 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e94815-cbed-47cf-ad06-6a4c5783d571-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:09:15.901433 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.901391 2569 generic.go:356] "Generic (PLEG): container finished" podID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerID="bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e" exitCode=0 Jul 05 14:09:15.901575 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.901442 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" Jul 05 14:09:15.901575 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.901439 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" event={"ID":"34e94815-cbed-47cf-ad06-6a4c5783d571","Type":"ContainerDied","Data":"bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e"} Jul 05 14:09:15.901575 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.901561 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr" event={"ID":"34e94815-cbed-47cf-ad06-6a4c5783d571","Type":"ContainerDied","Data":"cfdaa06c67686b4abc3dcfdf589c61e71e8191b9e00ccc40fc75df2ecdf5a0b3"} Jul 05 14:09:15.901696 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.901589 2569 scope.go:117] "RemoveContainer" containerID="bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e" Jul 05 14:09:15.910984 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.910966 2569 scope.go:117] "RemoveContainer" containerID="2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6" Jul 05 14:09:15.919275 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.919254 2569 scope.go:117] "RemoveContainer" containerID="bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e" Jul 05 14:09:15.919542 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:09:15.919523 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e\": container with ID starting with bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e not found: ID does not exist" containerID="bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e" Jul 05 14:09:15.919624 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.919553 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e"} err="failed to get container status \"bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e\": rpc error: code = NotFound desc = could not find container \"bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e\": container with ID starting with bec9d128d08f7d599a8b81e028ccc03a4fc88fd2ac196e588585dcacc6ccb27e not found: ID does not exist" Jul 05 14:09:15.919624 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.919576 2569 scope.go:117] "RemoveContainer" containerID="2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6" Jul 05 14:09:15.919808 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:09:15.919789 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6\": container with ID starting with 2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6 not found: ID does not exist" containerID="2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6" Jul 05 14:09:15.919847 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.919813 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6"} err="failed to get container status \"2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6\": rpc error: code = NotFound desc = could not find container \"2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6\": container with ID starting with 2251c22c680a7a39e1cc2a9e63f30564aeb51a53127f6486574084d8561a71d6 not found: ID does not exist" Jul 05 14:09:15.926314 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.926292 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr"] Jul 05 14:09:15.930966 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:15.930946 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-28ccab57-openshift-default-986b88495-x9wcr"] Jul 05 14:09:16.799617 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:16.799585 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" path="/var/lib/kubelet/pods/34e94815-cbed-47cf-ad06-6a4c5783d571/volumes" Jul 05 14:09:28.136165 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136123 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt"] Jul 05 14:09:28.136684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136573 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" Jul 05 14:09:28.136684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136590 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" Jul 05 14:09:28.136684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136610 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" Jul 05 14:09:28.136684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136615 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" Jul 05 14:09:28.136684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136675 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" Jul 05 14:09:28.136684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.136684 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="34e94815-cbed-47cf-ad06-6a4c5783d571" containerName="istio-proxy" Jul 05 14:09:28.139785 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.139764 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.142825 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.142800 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-2aab95e7-openshift-default-dockercfg-9gsdm\"" Jul 05 14:09:28.153254 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.153227 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt"] Jul 05 14:09:28.264151 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264112 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-credential-socket\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264151 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-envoy\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-socket\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264230 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-podinfo\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264251 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74deb992-b48d-4500-ae6e-8985e09aa1b0-istiod-ca-cert\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264299 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-certs\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264337 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-token\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264602 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-data\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.264602 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.264496 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8gv2\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-kube-api-access-n8gv2\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.365914 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.365874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-data\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.365927 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8gv2\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-kube-api-access-n8gv2\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.365953 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-credential-socket\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.365972 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-envoy\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.365991 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-socket\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366022 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-podinfo\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366037 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74deb992-b48d-4500-ae6e-8985e09aa1b0-istiod-ca-cert\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366519 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366172 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-certs\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366519 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-token\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366519 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-data\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366519 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366441 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-socket\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366519 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366479 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-credential-socket\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366790 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-certs\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.366971 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.366950 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74deb992-b48d-4500-ae6e-8985e09aa1b0-istiod-ca-cert\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.368390 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.368354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-envoy\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.368613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.368595 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-podinfo\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.375037 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.375013 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-token\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.375248 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.375229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8gv2\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-kube-api-access-n8gv2\") pod \"e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.451111 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.451079 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:28.588451 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.588423 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt"] Jul 05 14:09:28.591073 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:09:28.591044 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74deb992_b48d_4500_ae6e_8985e09aa1b0.slice/crio-aa84ac2497d8d7d91db07da824e7f31aeee651e2673b59061bda29e4d2dd8331 WatchSource:0}: Error finding container aa84ac2497d8d7d91db07da824e7f31aeee651e2673b59061bda29e4d2dd8331: Status 404 returned error can't find the container with id aa84ac2497d8d7d91db07da824e7f31aeee651e2673b59061bda29e4d2dd8331 Jul 05 14:09:28.593376 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.593324 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:28.593472 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.593425 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:28.593472 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.593460 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:28.970887 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.970846 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" event={"ID":"74deb992-b48d-4500-ae6e-8985e09aa1b0","Type":"ContainerStarted","Data":"620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d"} Jul 05 14:09:28.971101 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.970909 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" event={"ID":"74deb992-b48d-4500-ae6e-8985e09aa1b0","Type":"ContainerStarted","Data":"aa84ac2497d8d7d91db07da824e7f31aeee651e2673b59061bda29e4d2dd8331"} Jul 05 14:09:28.991080 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:28.991025 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podStartSLOduration=0.991009163 podStartE2EDuration="991.009163ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:09:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:09:28.990198134 +0000 UTC m=+1856.999667783" watchObservedRunningTime="2026-07-05 14:09:28.991009163 +0000 UTC m=+1857.000478783" Jul 05 14:09:29.451306 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:29.451271 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:29.456188 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:29.456161 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:29.975663 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:29.975627 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:29.976570 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:29.976545 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:44.977769 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:44.977729 2569 patch_prober.go:28] interesting pod/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 05 14:09:44.978204 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:44.977787 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 05 14:09:45.052621 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:45.052584 2569 generic.go:356] "Generic (PLEG): container finished" podID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerID="620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d" exitCode=0 Jul 05 14:09:45.052796 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:45.052653 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" event={"ID":"74deb992-b48d-4500-ae6e-8985e09aa1b0","Type":"ContainerDied","Data":"620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d"} Jul 05 14:09:45.053140 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:45.053116 2569 scope.go:117] "RemoveContainer" containerID="620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d" Jul 05 14:09:45.054257 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:45.054225 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:45.054346 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:45.054328 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:45.054403 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:45.054392 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:09:46.059239 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:46.059204 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" event={"ID":"74deb992-b48d-4500-ae6e-8985e09aa1b0","Type":"ContainerStarted","Data":"d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f"} Jul 05 14:09:46.451610 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:46.451578 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:46.451836 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:46.451815 2569 patch_prober.go:28] interesting pod/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 05 14:09:46.451882 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:46.451861 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 05 14:09:47.452259 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:47.452217 2569 patch_prober.go:28] interesting pod/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 05 14:09:47.452715 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:47.452283 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 05 14:09:48.452221 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:48.452179 2569 patch_prober.go:28] interesting pod/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 05 14:09:48.452424 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:48.452236 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 05 14:09:49.452445 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:49.452399 2569 patch_prober.go:28] interesting pod/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 05 14:09:49.452876 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:49.452477 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 05 14:09:50.455242 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:50.455215 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:50.455694 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:50.455491 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:09:50.456260 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:09:50.456244 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:10:08.995025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:08.994990 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz"] Jul 05 14:10:09.003177 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.003140 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.006451 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.006404 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2aab95e7\"/\"openshift-service-ca.crt\"" Jul 05 14:10:09.008097 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.008072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2aab95e7\"/\"default-dockercfg-gr4n4\"" Jul 05 14:10:09.008238 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.008122 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2aab95e7\"/\"kube-root-ca.crt\"" Jul 05 14:10:09.008238 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.008072 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2aab95e7\"/\"e2e-lifecycle-model-2aab95e7-kserve-self-signed-certs\"" Jul 05 14:10:09.010334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.010309 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz"] Jul 05 14:10:09.063070 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.063038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-tmp-dir\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.063249 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.063101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97gcn\" (UniqueName: \"kubernetes.io/projected/7915bc84-569b-465c-b340-6e8c060d0c74-kube-api-access-97gcn\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.063249 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.063136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-home\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.063249 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.063161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-model-cache\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.063249 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.063188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7915bc84-569b-465c-b340-6e8c060d0c74-tls-certs\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.063472 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.063292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-dshm\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.164689 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.164647 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-home\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.164907 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.164697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-model-cache\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.164907 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.164810 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7915bc84-569b-465c-b340-6e8c060d0c74-tls-certs\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.164907 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.164887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-dshm\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.165080 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.164938 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-tmp-dir\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.165080 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.165023 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97gcn\" (UniqueName: \"kubernetes.io/projected/7915bc84-569b-465c-b340-6e8c060d0c74-kube-api-access-97gcn\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.165189 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.165142 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-model-cache\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.165239 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.165182 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-home\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.165340 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.165321 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-tmp-dir\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.167166 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.167135 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-dshm\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.167500 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.167480 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7915bc84-569b-465c-b340-6e8c060d0c74-tls-certs\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.174576 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.174554 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97gcn\" (UniqueName: \"kubernetes.io/projected/7915bc84-569b-465c-b340-6e8c060d0c74-kube-api-access-97gcn\") pod \"e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.314124 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.314020 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:09.457724 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:09.457690 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz"] Jul 05 14:10:09.461377 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:10:09.461336 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7915bc84_569b_465c_b340_6e8c060d0c74.slice/crio-d54ed1f3b424a37da3f9db3940ba4f2dfe71631bcdd431fccb78432fdea15a6e WatchSource:0}: Error finding container d54ed1f3b424a37da3f9db3940ba4f2dfe71631bcdd431fccb78432fdea15a6e: Status 404 returned error can't find the container with id d54ed1f3b424a37da3f9db3940ba4f2dfe71631bcdd431fccb78432fdea15a6e Jul 05 14:10:10.178139 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:10.178105 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" event={"ID":"7915bc84-569b-465c-b340-6e8c060d0c74","Type":"ContainerStarted","Data":"5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d"} Jul 05 14:10:10.178530 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:10.178145 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" event={"ID":"7915bc84-569b-465c-b340-6e8c060d0c74","Type":"ContainerStarted","Data":"d54ed1f3b424a37da3f9db3940ba4f2dfe71631bcdd431fccb78432fdea15a6e"} Jul 05 14:10:10.198104 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:10.198057 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" podStartSLOduration=2.198044003 podStartE2EDuration="2.198044003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:10:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:10:10.197263548 +0000 UTC m=+1898.206733176" watchObservedRunningTime="2026-07-05 14:10:10.198044003 +0000 UTC m=+1898.207513931" Jul 05 14:10:19.314851 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:19.314814 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:19.315268 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:19.315069 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:19.327559 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:19.327532 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:20.234766 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:20.234737 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:39.450359 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:39.450316 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz"] Jul 05 14:10:54.471112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.471049 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" podUID="7915bc84-569b-465c-b340-6e8c060d0c74" containerName="main" containerID="cri-o://5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d" gracePeriod=45 Jul 05 14:10:54.726208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.726138 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:54.812331 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812303 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-home\") pod \"7915bc84-569b-465c-b340-6e8c060d0c74\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " Jul 05 14:10:54.812553 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812339 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-dshm\") pod \"7915bc84-569b-465c-b340-6e8c060d0c74\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " Jul 05 14:10:54.812553 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812356 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-model-cache\") pod \"7915bc84-569b-465c-b340-6e8c060d0c74\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " Jul 05 14:10:54.812553 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812444 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7915bc84-569b-465c-b340-6e8c060d0c74-tls-certs\") pod \"7915bc84-569b-465c-b340-6e8c060d0c74\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " Jul 05 14:10:54.812553 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812510 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97gcn\" (UniqueName: \"kubernetes.io/projected/7915bc84-569b-465c-b340-6e8c060d0c74-kube-api-access-97gcn\") pod \"7915bc84-569b-465c-b340-6e8c060d0c74\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " Jul 05 14:10:54.812790 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812566 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-tmp-dir\") pod \"7915bc84-569b-465c-b340-6e8c060d0c74\" (UID: \"7915bc84-569b-465c-b340-6e8c060d0c74\") " Jul 05 14:10:54.812790 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812615 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-home" (OuterVolumeSpecName: "home") pod "7915bc84-569b-465c-b340-6e8c060d0c74" (UID: "7915bc84-569b-465c-b340-6e8c060d0c74"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:10:54.812790 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812653 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-model-cache" (OuterVolumeSpecName: "model-cache") pod "7915bc84-569b-465c-b340-6e8c060d0c74" (UID: "7915bc84-569b-465c-b340-6e8c060d0c74"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:10:54.812917 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812854 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7915bc84-569b-465c-b340-6e8c060d0c74" (UID: "7915bc84-569b-465c-b340-6e8c060d0c74"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:10:54.813006 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.812983 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:10:54.813006 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.813006 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:10:54.813185 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.813016 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:10:54.814792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.814764 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-dshm" (OuterVolumeSpecName: "dshm") pod "7915bc84-569b-465c-b340-6e8c060d0c74" (UID: "7915bc84-569b-465c-b340-6e8c060d0c74"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:10:54.814886 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.814841 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7915bc84-569b-465c-b340-6e8c060d0c74-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7915bc84-569b-465c-b340-6e8c060d0c74" (UID: "7915bc84-569b-465c-b340-6e8c060d0c74"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:10:54.814886 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.814850 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7915bc84-569b-465c-b340-6e8c060d0c74-kube-api-access-97gcn" (OuterVolumeSpecName: "kube-api-access-97gcn") pod "7915bc84-569b-465c-b340-6e8c060d0c74" (UID: "7915bc84-569b-465c-b340-6e8c060d0c74"). InnerVolumeSpecName "kube-api-access-97gcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:10:54.914146 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.914107 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7915bc84-569b-465c-b340-6e8c060d0c74-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:10:54.914146 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.914137 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97gcn\" (UniqueName: \"kubernetes.io/projected/7915bc84-569b-465c-b340-6e8c060d0c74-kube-api-access-97gcn\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:10:54.914146 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:54.914147 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7915bc84-569b-465c-b340-6e8c060d0c74-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:10:55.393480 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.393444 2569 generic.go:356] "Generic (PLEG): container finished" podID="7915bc84-569b-465c-b340-6e8c060d0c74" containerID="5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d" exitCode=0 Jul 05 14:10:55.393659 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.393511 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" Jul 05 14:10:55.393659 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.393527 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" event={"ID":"7915bc84-569b-465c-b340-6e8c060d0c74","Type":"ContainerDied","Data":"5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d"} Jul 05 14:10:55.393659 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.393574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz" event={"ID":"7915bc84-569b-465c-b340-6e8c060d0c74","Type":"ContainerDied","Data":"d54ed1f3b424a37da3f9db3940ba4f2dfe71631bcdd431fccb78432fdea15a6e"} Jul 05 14:10:55.393659 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.393591 2569 scope.go:117] "RemoveContainer" containerID="5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d" Jul 05 14:10:55.402693 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.402669 2569 scope.go:117] "RemoveContainer" containerID="5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d" Jul 05 14:10:55.402988 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:10:55.402965 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d\": container with ID starting with 5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d not found: ID does not exist" containerID="5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d" Jul 05 14:10:55.403057 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.403000 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d"} err="failed to get container status \"5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d\": rpc error: code = NotFound desc = could not find container \"5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d\": container with ID starting with 5cd64281de97194acbc2865078915e1afa12e9bf6a86672d7218d252b4b12e0d not found: ID does not exist" Jul 05 14:10:55.415581 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.415554 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz"] Jul 05 14:10:55.419607 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:55.419582 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2aab95e7/e2e-lifecycle-model-2aab95e7-kserve-696f4cf579-bcqtz"] Jul 05 14:10:56.799580 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:10:56.799547 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7915bc84-569b-465c-b340-6e8c060d0c74" path="/var/lib/kubelet/pods/7915bc84-569b-465c-b340-6e8c060d0c74/volumes" Jul 05 14:11:38.531201 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:38.531168 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt"] Jul 05 14:11:38.531768 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:38.531408 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" containerID="cri-o://d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f" gracePeriod=30 Jul 05 14:11:43.792521 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.792454 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:11:43.885914 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.885881 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-certs\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886106 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.885933 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-envoy\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886106 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.885959 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-token\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886106 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.885982 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74deb992-b48d-4500-ae6e-8985e09aa1b0-istiod-ca-cert\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886106 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886041 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-podinfo\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886116 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-socket\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886139 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-data\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886171 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8gv2\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-kube-api-access-n8gv2\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886195 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-credential-socket\") pod \"74deb992-b48d-4500-ae6e-8985e09aa1b0\" (UID: \"74deb992-b48d-4500-ae6e-8985e09aa1b0\") " Jul 05 14:11:43.886334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886236 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:11:43.886648 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886455 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:11:43.886648 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886515 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74deb992-b48d-4500-ae6e-8985e09aa1b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:11:43.886648 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886608 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74deb992-b48d-4500-ae6e-8985e09aa1b0-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.886648 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886629 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.886648 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886642 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.886899 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.886707 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:11:43.887064 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.887043 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:11:43.888333 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.888311 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:11:43.888333 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.888318 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:11:43.888704 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.888673 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:11:43.888761 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.888713 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-kube-api-access-n8gv2" (OuterVolumeSpecName: "kube-api-access-n8gv2") pod "74deb992-b48d-4500-ae6e-8985e09aa1b0" (UID: "74deb992-b48d-4500-ae6e-8985e09aa1b0"). InnerVolumeSpecName "kube-api-access-n8gv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:11:43.987534 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.987498 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.987534 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.987529 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.987534 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.987538 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.987777 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.987549 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.987777 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.987558 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8gv2\" (UniqueName: \"kubernetes.io/projected/74deb992-b48d-4500-ae6e-8985e09aa1b0-kube-api-access-n8gv2\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:43.987777 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:43.987568 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74deb992-b48d-4500-ae6e-8985e09aa1b0-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:11:44.625612 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.625578 2569 generic.go:356] "Generic (PLEG): container finished" podID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerID="d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f" exitCode=0 Jul 05 14:11:44.625792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.625647 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" Jul 05 14:11:44.625792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.625658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" event={"ID":"74deb992-b48d-4500-ae6e-8985e09aa1b0","Type":"ContainerDied","Data":"d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f"} Jul 05 14:11:44.625792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.625732 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt" event={"ID":"74deb992-b48d-4500-ae6e-8985e09aa1b0","Type":"ContainerDied","Data":"aa84ac2497d8d7d91db07da824e7f31aeee651e2673b59061bda29e4d2dd8331"} Jul 05 14:11:44.625792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.625754 2569 scope.go:117] "RemoveContainer" containerID="d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f" Jul 05 14:11:44.634941 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.634922 2569 scope.go:117] "RemoveContainer" containerID="620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d" Jul 05 14:11:44.642897 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.642879 2569 scope.go:117] "RemoveContainer" containerID="d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f" Jul 05 14:11:44.643148 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:11:44.643126 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f\": container with ID starting with d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f not found: ID does not exist" containerID="d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f" Jul 05 14:11:44.643214 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.643158 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f"} err="failed to get container status \"d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f\": rpc error: code = NotFound desc = could not find container \"d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f\": container with ID starting with d501920c5e8b04847c148fc2804ce301fd384b5e4c4c359b22199c071774b24f not found: ID does not exist" Jul 05 14:11:44.643214 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.643176 2569 scope.go:117] "RemoveContainer" containerID="620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d" Jul 05 14:11:44.643409 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:11:44.643392 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d\": container with ID starting with 620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d not found: ID does not exist" containerID="620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d" Jul 05 14:11:44.643459 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.643418 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d"} err="failed to get container status \"620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d\": rpc error: code = NotFound desc = could not find container \"620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d\": container with ID starting with 620d3c34680c390af532f5d344b1218ee71b1701f62d1b5be88d621698c0157d not found: ID does not exist" Jul 05 14:11:44.664701 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.664677 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt"] Jul 05 14:11:44.676528 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.676502 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-2aab95e7-openshift-default-59c648c68c-22jvt"] Jul 05 14:11:44.799271 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:11:44.799232 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" path="/var/lib/kubelet/pods/74deb992-b48d-4500-ae6e-8985e09aa1b0/volumes" Jul 05 14:12:35.050375 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050342 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh"] Jul 05 14:12:35.050780 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050718 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" Jul 05 14:12:35.050780 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050731 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" Jul 05 14:12:35.050780 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050776 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" Jul 05 14:12:35.050888 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050783 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" Jul 05 14:12:35.050888 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050799 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7915bc84-569b-465c-b340-6e8c060d0c74" containerName="main" Jul 05 14:12:35.050888 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050806 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="7915bc84-569b-465c-b340-6e8c060d0c74" containerName="main" Jul 05 14:12:35.050988 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050913 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" Jul 05 14:12:35.050988 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050931 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="7915bc84-569b-465c-b340-6e8c060d0c74" containerName="main" Jul 05 14:12:35.050988 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.050939 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="74deb992-b48d-4500-ae6e-8985e09aa1b0" containerName="istio-proxy" Jul 05 14:12:35.055311 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.055291 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.060299 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.060274 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-2d7b22-openshift-default-dockercfg-pck2v\"" Jul 05 14:12:35.066241 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.066219 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh"] Jul 05 14:12:35.066773 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.066753 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-certs\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.066852 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.066782 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-envoy\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.066891 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.066862 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-socket\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.066927 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.066900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-token\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.066975 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.066937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48f4r\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-kube-api-access-48f4r\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.067048 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.067033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-credential-socket\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.067103 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.067061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-data\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.067158 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.067106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istiod-ca-cert\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.067158 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.067135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-podinfo\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168287 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168244 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-certs\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168287 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168300 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-envoy\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168568 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-socket\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168568 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-token\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168568 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48f4r\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-kube-api-access-48f4r\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168613 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-credential-socket\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168637 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-data\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-certs\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168666 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istiod-ca-cert\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168746 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-socket\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.168805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168786 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-podinfo\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.169041 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.168993 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-data\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.169257 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.169234 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istiod-ca-cert\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.169393 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.169280 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-credential-socket\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.170685 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.170665 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-envoy\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.171059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.171038 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-podinfo\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.177413 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.177395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-token\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.177827 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.177806 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48f4r\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-kube-api-access-48f4r\") pod \"e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.366107 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.366003 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:35.506219 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.506194 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh"] Jul 05 14:12:35.508564 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:12:35.508533 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod855c91dc_f1ed_4e6d_87c5_e045936e5c3e.slice/crio-cf94e68c00e3973ce7f9b2873728a5d10068e3ac451a1231b1dee44d3a5a9b11 WatchSource:0}: Error finding container cf94e68c00e3973ce7f9b2873728a5d10068e3ac451a1231b1dee44d3a5a9b11: Status 404 returned error can't find the container with id cf94e68c00e3973ce7f9b2873728a5d10068e3ac451a1231b1dee44d3a5a9b11 Jul 05 14:12:35.510945 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.510909 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:35.511056 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.510991 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:35.511056 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.511038 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:35.865985 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.865943 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" event={"ID":"855c91dc-f1ed-4e6d-87c5-e045936e5c3e","Type":"ContainerStarted","Data":"e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb"} Jul 05 14:12:35.866173 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.866005 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" event={"ID":"855c91dc-f1ed-4e6d-87c5-e045936e5c3e","Type":"ContainerStarted","Data":"cf94e68c00e3973ce7f9b2873728a5d10068e3ac451a1231b1dee44d3a5a9b11"} Jul 05 14:12:35.891789 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:35.891737 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podStartSLOduration=0.89172291 podStartE2EDuration="891.72291ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:12:35.890435521 +0000 UTC m=+2043.899905178" watchObservedRunningTime="2026-07-05 14:12:35.89172291 +0000 UTC m=+2043.901192529" Jul 05 14:12:36.366745 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:36.366706 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:36.371466 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:36.371441 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:36.870441 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:36.870411 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:36.871259 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:36.871238 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:43.023974 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.023934 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh"] Jul 05 14:12:43.027504 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.027482 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.030580 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.030555 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7c0371-openshift-default-dockercfg-wxhls\"" Jul 05 14:12:43.041074 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.041051 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh"] Jul 05 14:12:43.137934 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.137900 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mtb4\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-kube-api-access-4mtb4\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138103 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.137948 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-data\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138103 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.137968 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-certs\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138103 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.138029 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istiod-ca-cert\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138103 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.138067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-podinfo\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138103 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.138096 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-credential-socket\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138357 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.138209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-token\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138357 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.138252 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-envoy\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.138357 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.138281 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-socket\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.238850 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.238821 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-token\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.238850 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.238856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-envoy\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.238879 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-socket\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.238964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mtb4\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-kube-api-access-4mtb4\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239005 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-data\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-certs\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239060 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istiod-ca-cert\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239340 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239083 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-podinfo\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239340 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239176 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-credential-socket\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239481 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-socket\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239481 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239434 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-certs\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239592 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239519 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-data\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239674 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239653 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-credential-socket\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.239812 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.239786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istiod-ca-cert\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.241227 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.241201 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-envoy\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.241591 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.241574 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-podinfo\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.247854 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.247829 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-token\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.247939 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.247870 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mtb4\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-kube-api-access-4mtb4\") pod \"e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.339138 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.339059 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:43.473342 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.473312 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh"] Jul 05 14:12:43.475402 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:12:43.475357 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86465ce5_eb39_45fe_8fd3_6fea44ddbdfa.slice/crio-48fd44611c4ed29e05b7ef28bd4359831e5e9fa0e854ddc9610115c90e7cbf6a WatchSource:0}: Error finding container 48fd44611c4ed29e05b7ef28bd4359831e5e9fa0e854ddc9610115c90e7cbf6a: Status 404 returned error can't find the container with id 48fd44611c4ed29e05b7ef28bd4359831e5e9fa0e854ddc9610115c90e7cbf6a Jul 05 14:12:43.477564 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.477531 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:43.477667 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.477601 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:43.477667 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.477631 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:43.903815 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.903778 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" event={"ID":"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa","Type":"ContainerStarted","Data":"205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1"} Jul 05 14:12:43.904038 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.903867 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" event={"ID":"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa","Type":"ContainerStarted","Data":"48fd44611c4ed29e05b7ef28bd4359831e5e9fa0e854ddc9610115c90e7cbf6a"} Jul 05 14:12:43.927691 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:43.927642 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podStartSLOduration=0.92762702 podStartE2EDuration="927.62702ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:12:43.925650237 +0000 UTC m=+2051.935119902" watchObservedRunningTime="2026-07-05 14:12:43.92762702 +0000 UTC m=+2051.937096640" Jul 05 14:12:44.340020 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:44.339978 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:44.344983 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:44.344955 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:44.909049 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:44.909011 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:44.910088 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:44.910068 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:12:52.912554 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:52.912508 2569 patch_prober.go:28] interesting pod/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 05 14:12:52.912962 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:52.912596 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 05 14:12:53.963073 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:53.963005 2569 generic.go:356] "Generic (PLEG): container finished" podID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerID="e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb" exitCode=0 Jul 05 14:12:53.963503 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:53.963078 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" event={"ID":"855c91dc-f1ed-4e6d-87c5-e045936e5c3e","Type":"ContainerDied","Data":"e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb"} Jul 05 14:12:53.963551 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:53.963506 2569 scope.go:117] "RemoveContainer" containerID="e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb" Jul 05 14:12:53.964527 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:53.964496 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:53.964600 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:53.964562 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:53.964644 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:53.964597 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:12:54.972655 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:54.972614 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" event={"ID":"855c91dc-f1ed-4e6d-87c5-e045936e5c3e","Type":"ContainerStarted","Data":"4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d"} Jul 05 14:12:55.368678 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:55.368578 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:55.369120 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:55.369092 2569 patch_prober.go:28] interesting pod/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 05 14:12:55.369312 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:55.369284 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 05 14:12:56.367391 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:56.366502 2569 patch_prober.go:28] interesting pod/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 05 14:12:56.367391 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:56.366567 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 05 14:12:57.366446 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:57.366322 2569 patch_prober.go:28] interesting pod/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 05 14:12:57.366446 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:57.366407 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 05 14:12:58.367488 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:58.367445 2569 patch_prober.go:28] interesting pod/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 05 14:12:58.367904 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:58.367509 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 05 14:12:59.370608 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:59.370574 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:59.371025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:59.370935 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:12:59.371536 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:12:59.371517 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:13:00.164180 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:00.163843 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:52328->10.132.0.43:15021: read: connection reset by peer" start-of-body= Jul 05 14:13:00.164180 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:00.163958 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": read tcp 10.132.0.2:52328->10.132.0.43:15021: read: connection reset by peer" Jul 05 14:13:01.004347 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:01.004317 2569 generic.go:356] "Generic (PLEG): container finished" podID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerID="205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1" exitCode=0 Jul 05 14:13:01.004853 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:01.004410 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" event={"ID":"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa","Type":"ContainerDied","Data":"205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1"} Jul 05 14:13:01.004927 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:01.004853 2569 scope.go:117] "RemoveContainer" containerID="205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1" Jul 05 14:13:01.005875 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:01.005844 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:13:01.005980 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:01.005934 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:13:01.006043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:01.005980 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:13:02.010922 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:02.010887 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" event={"ID":"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa","Type":"ContainerStarted","Data":"fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64"} Jul 05 14:13:02.339808 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:02.339697 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:13:02.340074 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:02.340037 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 05 14:13:02.340226 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:02.340099 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 05 14:13:03.340459 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:03.340421 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 05 14:13:03.341094 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:03.341060 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 05 14:13:04.340352 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.340312 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 05 14:13:04.340696 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.340408 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 05 14:13:04.491030 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.490999 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7"] Jul 05 14:13:04.493568 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.493549 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.498207 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.498183 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d7b22\"/\"kube-root-ca.crt\"" Jul 05 14:13:04.498334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.498239 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d7b22\"/\"default-dockercfg-rmd8s\"" Jul 05 14:13:04.498334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.498239 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-2d7b22\"/\"openshift-service-ca.crt\"" Jul 05 14:13:04.498334 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.498288 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-2d7b22\"/\"test-model-2d7b22-kserve-self-signed-certs\"" Jul 05 14:13:04.504491 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.504466 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7"] Jul 05 14:13:04.662061 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.661956 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-tmp-dir\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.662061 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.662010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-model-cache\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.662289 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.662108 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z777p\" (UniqueName: \"kubernetes.io/projected/81d6088a-78e6-4fef-b1d0-9075c422be03-kube-api-access-z777p\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.662289 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.662151 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-home\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.662289 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.662182 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d6088a-78e6-4fef-b1d0-9075c422be03-tls-certs\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.662289 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.662237 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-dshm\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.762828 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.762795 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-tmp-dir\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.762846 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-model-cache\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.762877 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z777p\" (UniqueName: \"kubernetes.io/projected/81d6088a-78e6-4fef-b1d0-9075c422be03-kube-api-access-z777p\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.762897 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-home\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.762915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d6088a-78e6-4fef-b1d0-9075c422be03-tls-certs\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763248 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.763226 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-tmp-dir\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763294 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.763241 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-dshm\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763294 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.763236 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-home\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.763386 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.763330 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-model-cache\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.765308 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.765287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-dshm\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.765456 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.765321 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d6088a-78e6-4fef-b1d0-9075c422be03-tls-certs\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.772562 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.772540 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z777p\" (UniqueName: \"kubernetes.io/projected/81d6088a-78e6-4fef-b1d0-9075c422be03-kube-api-access-z777p\") pod \"test-model-2d7b22-kserve-74d8b9f6cf-cvnj7\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.804339 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.804317 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:04.985552 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:04.985528 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7"] Jul 05 14:13:04.987675 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:13:04.987650 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81d6088a_78e6_4fef_b1d0_9075c422be03.slice/crio-d00bb32992e977384087e6394de43abdacb9e39b3798066e46f113d586c712e6 WatchSource:0}: Error finding container d00bb32992e977384087e6394de43abdacb9e39b3798066e46f113d586c712e6: Status 404 returned error can't find the container with id d00bb32992e977384087e6394de43abdacb9e39b3798066e46f113d586c712e6 Jul 05 14:13:05.046772 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:05.046731 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" event={"ID":"81d6088a-78e6-4fef-b1d0-9075c422be03","Type":"ContainerStarted","Data":"d00bb32992e977384087e6394de43abdacb9e39b3798066e46f113d586c712e6"} Jul 05 14:13:05.340974 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:05.340464 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 05 14:13:05.340974 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:05.340529 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 05 14:13:06.052531 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:06.052496 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" event={"ID":"81d6088a-78e6-4fef-b1d0-9075c422be03","Type":"ContainerStarted","Data":"579ce5a12dae7ccd676f7d163f26c8cbdd18fc1d4e82107757149537d77bf6c6"} Jul 05 14:13:06.071888 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:06.071814 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" podStartSLOduration=2.071795421 podStartE2EDuration="2.071795421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:13:06.071573289 +0000 UTC m=+2074.081042933" watchObservedRunningTime="2026-07-05 14:13:06.071795421 +0000 UTC m=+2074.081265041" Jul 05 14:13:06.343147 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:06.343067 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:13:06.343573 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:06.343319 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:13:06.344100 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:06.344085 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:13:14.805421 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:14.805391 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:14.805421 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:14.805428 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:14.817755 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:14.817732 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:15.110438 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:15.110347 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:13:32.770303 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.770267 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh"] Jul 05 14:13:32.775656 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.775637 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:32.780140 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.780016 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7c0371\"/\"openshift-service-ca.crt\"" Jul 05 14:13:32.780140 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.780135 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7c0371\"/\"default-dockercfg-b2c7t\"" Jul 05 14:13:32.780319 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.780022 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7c0371\"/\"kube-root-ca.crt\"" Jul 05 14:13:32.780439 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.780421 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7c0371\"/\"test-model-7c0371-kserve-self-signed-certs\"" Jul 05 14:13:32.782543 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.782523 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh"] Jul 05 14:13:32.921828 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.921795 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-tmp-dir\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:32.922004 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.921834 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x442g\" (UniqueName: \"kubernetes.io/projected/ebd2010f-f780-4575-99f9-70c2e2313773-kube-api-access-x442g\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:32.922004 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.921857 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-model-cache\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:32.922004 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.921895 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd2010f-f780-4575-99f9-70c2e2313773-tls-certs\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:32.922004 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.921955 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-home\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:32.922004 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:32.921991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-dshm\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.022928 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.022835 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-model-cache\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.022928 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.022887 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd2010f-f780-4575-99f9-70c2e2313773-tls-certs\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023118 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023062 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-home\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023215 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-dshm\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023294 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-model-cache\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023356 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023345 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-tmp-dir\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023432 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023394 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-home\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023432 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023398 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x442g\" (UniqueName: \"kubernetes.io/projected/ebd2010f-f780-4575-99f9-70c2e2313773-kube-api-access-x442g\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.023696 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.023676 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-tmp-dir\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.025263 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.025240 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-dshm\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.025464 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.025447 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd2010f-f780-4575-99f9-70c2e2313773-tls-certs\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.032303 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.032280 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x442g\" (UniqueName: \"kubernetes.io/projected/ebd2010f-f780-4575-99f9-70c2e2313773-kube-api-access-x442g\") pod \"test-model-7c0371-kserve-9f4b9cc4c-c7hkh\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.087485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.087457 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:33.219036 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:33.219010 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh"] Jul 05 14:13:33.220765 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:13:33.220738 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podebd2010f_f780_4575_99f9_70c2e2313773.slice/crio-5103ee365cc3ed817e56dc465d99005a60fed7fba80775342ab48db721610a7a WatchSource:0}: Error finding container 5103ee365cc3ed817e56dc465d99005a60fed7fba80775342ab48db721610a7a: Status 404 returned error can't find the container with id 5103ee365cc3ed817e56dc465d99005a60fed7fba80775342ab48db721610a7a Jul 05 14:13:34.188953 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:34.188914 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" event={"ID":"ebd2010f-f780-4575-99f9-70c2e2313773","Type":"ContainerStarted","Data":"77e46ec1f4d95c01b09cd3d416dff4a3413ef3a105a32055bd896e2a1d9cba67"} Jul 05 14:13:34.189387 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:34.188964 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" event={"ID":"ebd2010f-f780-4575-99f9-70c2e2313773","Type":"ContainerStarted","Data":"5103ee365cc3ed817e56dc465d99005a60fed7fba80775342ab48db721610a7a"} Jul 05 14:13:34.212357 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:34.212311 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" podStartSLOduration=2.21229698 podStartE2EDuration="2.21229698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:13:34.21065195 +0000 UTC m=+2102.220121570" watchObservedRunningTime="2026-07-05 14:13:34.21229698 +0000 UTC m=+2102.221766599" Jul 05 14:13:43.087805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:43.087765 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:43.088250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:43.087847 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:43.100478 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:43.100451 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:43.245757 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:43.245727 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:13:59.184486 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:13:59.184453 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7"] Jul 05 14:14:00.195459 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:00.195429 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh"] Jul 05 14:14:14.204715 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.204672 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" podUID="81d6088a-78e6-4fef-b1d0-9075c422be03" containerName="main" containerID="cri-o://579ce5a12dae7ccd676f7d163f26c8cbdd18fc1d4e82107757149537d77bf6c6" gracePeriod=45 Jul 05 14:14:14.389919 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.389880 2569 generic.go:356] "Generic (PLEG): container finished" podID="81d6088a-78e6-4fef-b1d0-9075c422be03" containerID="579ce5a12dae7ccd676f7d163f26c8cbdd18fc1d4e82107757149537d77bf6c6" exitCode=0 Jul 05 14:14:14.390069 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.389959 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" event={"ID":"81d6088a-78e6-4fef-b1d0-9075c422be03","Type":"ContainerDied","Data":"579ce5a12dae7ccd676f7d163f26c8cbdd18fc1d4e82107757149537d77bf6c6"} Jul 05 14:14:14.458678 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.458603 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:14:14.615846 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.615818 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-dshm\") pod \"81d6088a-78e6-4fef-b1d0-9075c422be03\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " Jul 05 14:14:14.616051 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.615893 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-model-cache\") pod \"81d6088a-78e6-4fef-b1d0-9075c422be03\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " Jul 05 14:14:14.616051 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.615938 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d6088a-78e6-4fef-b1d0-9075c422be03-tls-certs\") pod \"81d6088a-78e6-4fef-b1d0-9075c422be03\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " Jul 05 14:14:14.616051 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616032 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-tmp-dir\") pod \"81d6088a-78e6-4fef-b1d0-9075c422be03\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " Jul 05 14:14:14.616223 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616066 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-home\") pod \"81d6088a-78e6-4fef-b1d0-9075c422be03\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " Jul 05 14:14:14.616223 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616132 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z777p\" (UniqueName: \"kubernetes.io/projected/81d6088a-78e6-4fef-b1d0-9075c422be03-kube-api-access-z777p\") pod \"81d6088a-78e6-4fef-b1d0-9075c422be03\" (UID: \"81d6088a-78e6-4fef-b1d0-9075c422be03\") " Jul 05 14:14:14.616223 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616158 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-model-cache" (OuterVolumeSpecName: "model-cache") pod "81d6088a-78e6-4fef-b1d0-9075c422be03" (UID: "81d6088a-78e6-4fef-b1d0-9075c422be03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:14.616469 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616282 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "81d6088a-78e6-4fef-b1d0-9075c422be03" (UID: "81d6088a-78e6-4fef-b1d0-9075c422be03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:14.616469 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616347 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-home" (OuterVolumeSpecName: "home") pod "81d6088a-78e6-4fef-b1d0-9075c422be03" (UID: "81d6088a-78e6-4fef-b1d0-9075c422be03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:14.616590 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616499 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:14.616590 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616511 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:14.616590 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.616519 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:14.618149 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.618128 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-dshm" (OuterVolumeSpecName: "dshm") pod "81d6088a-78e6-4fef-b1d0-9075c422be03" (UID: "81d6088a-78e6-4fef-b1d0-9075c422be03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:14.618228 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.618209 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d6088a-78e6-4fef-b1d0-9075c422be03-kube-api-access-z777p" (OuterVolumeSpecName: "kube-api-access-z777p") pod "81d6088a-78e6-4fef-b1d0-9075c422be03" (UID: "81d6088a-78e6-4fef-b1d0-9075c422be03"). InnerVolumeSpecName "kube-api-access-z777p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:14:14.618273 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.618234 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81d6088a-78e6-4fef-b1d0-9075c422be03-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "81d6088a-78e6-4fef-b1d0-9075c422be03" (UID: "81d6088a-78e6-4fef-b1d0-9075c422be03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:14:14.717059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.716976 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z777p\" (UniqueName: \"kubernetes.io/projected/81d6088a-78e6-4fef-b1d0-9075c422be03-kube-api-access-z777p\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:14.717059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.717012 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81d6088a-78e6-4fef-b1d0-9075c422be03-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:14.717059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:14.717023 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81d6088a-78e6-4fef-b1d0-9075c422be03-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:15.216968 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.216930 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" podUID="ebd2010f-f780-4575-99f9-70c2e2313773" containerName="main" containerID="cri-o://77e46ec1f4d95c01b09cd3d416dff4a3413ef3a105a32055bd896e2a1d9cba67" gracePeriod=45 Jul 05 14:14:15.398098 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.398062 2569 generic.go:356] "Generic (PLEG): container finished" podID="ebd2010f-f780-4575-99f9-70c2e2313773" containerID="77e46ec1f4d95c01b09cd3d416dff4a3413ef3a105a32055bd896e2a1d9cba67" exitCode=0 Jul 05 14:14:15.398225 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.398132 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" event={"ID":"ebd2010f-f780-4575-99f9-70c2e2313773","Type":"ContainerDied","Data":"77e46ec1f4d95c01b09cd3d416dff4a3413ef3a105a32055bd896e2a1d9cba67"} Jul 05 14:14:15.399558 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.399535 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" event={"ID":"81d6088a-78e6-4fef-b1d0-9075c422be03","Type":"ContainerDied","Data":"d00bb32992e977384087e6394de43abdacb9e39b3798066e46f113d586c712e6"} Jul 05 14:14:15.399702 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.399572 2569 scope.go:117] "RemoveContainer" containerID="579ce5a12dae7ccd676f7d163f26c8cbdd18fc1d4e82107757149537d77bf6c6" Jul 05 14:14:15.399702 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.399588 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7" Jul 05 14:14:15.420845 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.420821 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7"] Jul 05 14:14:15.424479 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.424456 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-2d7b22/test-model-2d7b22-kserve-74d8b9f6cf-cvnj7"] Jul 05 14:14:15.477002 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.476946 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:14:15.624930 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.624893 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x442g\" (UniqueName: \"kubernetes.io/projected/ebd2010f-f780-4575-99f9-70c2e2313773-kube-api-access-x442g\") pod \"ebd2010f-f780-4575-99f9-70c2e2313773\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " Jul 05 14:14:15.624930 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.624936 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-model-cache\") pod \"ebd2010f-f780-4575-99f9-70c2e2313773\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " Jul 05 14:14:15.625183 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625004 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd2010f-f780-4575-99f9-70c2e2313773-tls-certs\") pod \"ebd2010f-f780-4575-99f9-70c2e2313773\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " Jul 05 14:14:15.625183 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625087 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-home\") pod \"ebd2010f-f780-4575-99f9-70c2e2313773\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " Jul 05 14:14:15.625183 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625157 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-tmp-dir\") pod \"ebd2010f-f780-4575-99f9-70c2e2313773\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " Jul 05 14:14:15.625333 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625189 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-dshm\") pod \"ebd2010f-f780-4575-99f9-70c2e2313773\" (UID: \"ebd2010f-f780-4575-99f9-70c2e2313773\") " Jul 05 14:14:15.625333 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625238 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-model-cache" (OuterVolumeSpecName: "model-cache") pod "ebd2010f-f780-4575-99f9-70c2e2313773" (UID: "ebd2010f-f780-4575-99f9-70c2e2313773"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:15.625462 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625388 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ebd2010f-f780-4575-99f9-70c2e2313773" (UID: "ebd2010f-f780-4575-99f9-70c2e2313773"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:15.625462 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625424 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-home" (OuterVolumeSpecName: "home") pod "ebd2010f-f780-4575-99f9-70c2e2313773" (UID: "ebd2010f-f780-4575-99f9-70c2e2313773"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:15.625617 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625594 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:15.625682 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625618 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:15.625682 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.625627 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:15.627320 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.627298 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-dshm" (OuterVolumeSpecName: "dshm") pod "ebd2010f-f780-4575-99f9-70c2e2313773" (UID: "ebd2010f-f780-4575-99f9-70c2e2313773"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:15.627531 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.627430 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebd2010f-f780-4575-99f9-70c2e2313773-kube-api-access-x442g" (OuterVolumeSpecName: "kube-api-access-x442g") pod "ebd2010f-f780-4575-99f9-70c2e2313773" (UID: "ebd2010f-f780-4575-99f9-70c2e2313773"). InnerVolumeSpecName "kube-api-access-x442g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:14:15.627531 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.627473 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebd2010f-f780-4575-99f9-70c2e2313773-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ebd2010f-f780-4575-99f9-70c2e2313773" (UID: "ebd2010f-f780-4575-99f9-70c2e2313773"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:14:15.726747 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.726702 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ebd2010f-f780-4575-99f9-70c2e2313773-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:15.726747 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.726742 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x442g\" (UniqueName: \"kubernetes.io/projected/ebd2010f-f780-4575-99f9-70c2e2313773-kube-api-access-x442g\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:15.726747 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:15.726753 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ebd2010f-f780-4575-99f9-70c2e2313773-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:16.406045 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.406018 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" Jul 05 14:14:16.406526 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.406051 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh" event={"ID":"ebd2010f-f780-4575-99f9-70c2e2313773","Type":"ContainerDied","Data":"5103ee365cc3ed817e56dc465d99005a60fed7fba80775342ab48db721610a7a"} Jul 05 14:14:16.406526 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.406103 2569 scope.go:117] "RemoveContainer" containerID="77e46ec1f4d95c01b09cd3d416dff4a3413ef3a105a32055bd896e2a1d9cba67" Jul 05 14:14:16.428224 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.428200 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh"] Jul 05 14:14:16.432049 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.432029 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7c0371/test-model-7c0371-kserve-9f4b9cc4c-c7hkh"] Jul 05 14:14:16.799813 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.799779 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81d6088a-78e6-4fef-b1d0-9075c422be03" path="/var/lib/kubelet/pods/81d6088a-78e6-4fef-b1d0-9075c422be03/volumes" Jul 05 14:14:16.800192 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:16.800178 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd2010f-f780-4575-99f9-70c2e2313773" path="/var/lib/kubelet/pods/ebd2010f-f780-4575-99f9-70c2e2313773/volumes" Jul 05 14:14:43.084743 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:43.084709 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh"] Jul 05 14:14:43.085308 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:43.084967 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" containerID="cri-o://4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d" gracePeriod=30 Jul 05 14:14:44.373594 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:44.373561 2569 patch_prober.go:28] interesting pod/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 05 14:14:44.374005 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:44.373650 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 14:14:48.338486 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.338462 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:14:48.421024 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.420957 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-data\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421024 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421006 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-certs\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421037 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istiod-ca-cert\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421059 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-envoy\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421075 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-48f4r\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-kube-api-access-48f4r\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421142 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-token\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421169 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-credential-socket\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421184 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-podinfo\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421205 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-socket\") pod \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\" (UID: \"855c91dc-f1ed-4e6d-87c5-e045936e5c3e\") " Jul 05 14:14:48.421605 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421350 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:48.421605 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421578 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:48.421711 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421518 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:14:48.421711 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421644 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.421792 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421729 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:48.421860 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.421833 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-data" (OuterVolumeSpecName: "istio-data") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:48.423233 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.423207 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:48.423348 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.423308 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-token" (OuterVolumeSpecName: "istio-token") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:14:48.423492 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.423460 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:14:48.423582 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.423510 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-kube-api-access-48f4r" (OuterVolumeSpecName: "kube-api-access-48f4r") pod "855c91dc-f1ed-4e6d-87c5-e045936e5c3e" (UID: "855c91dc-f1ed-4e6d-87c5-e045936e5c3e"). InnerVolumeSpecName "kube-api-access-48f4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:14:48.522734 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522707 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522734 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522729 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522734 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522739 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48f4r\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-kube-api-access-48f4r\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522915 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522748 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522915 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522758 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522915 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522766 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522915 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522774 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.522915 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.522782 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/855c91dc-f1ed-4e6d-87c5-e045936e5c3e-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:48.563262 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.563227 2569 generic.go:356] "Generic (PLEG): container finished" podID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerID="4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d" exitCode=0 Jul 05 14:14:48.563353 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.563276 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" Jul 05 14:14:48.563353 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.563296 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" event={"ID":"855c91dc-f1ed-4e6d-87c5-e045936e5c3e","Type":"ContainerDied","Data":"4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d"} Jul 05 14:14:48.563353 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.563338 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh" event={"ID":"855c91dc-f1ed-4e6d-87c5-e045936e5c3e","Type":"ContainerDied","Data":"cf94e68c00e3973ce7f9b2873728a5d10068e3ac451a1231b1dee44d3a5a9b11"} Jul 05 14:14:48.563508 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.563359 2569 scope.go:117] "RemoveContainer" containerID="4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d" Jul 05 14:14:48.572493 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.572477 2569 scope.go:117] "RemoveContainer" containerID="e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb" Jul 05 14:14:48.580456 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.580440 2569 scope.go:117] "RemoveContainer" containerID="4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d" Jul 05 14:14:48.580685 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:14:48.580663 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d\": container with ID starting with 4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d not found: ID does not exist" containerID="4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d" Jul 05 14:14:48.580769 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.580690 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d"} err="failed to get container status \"4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d\": rpc error: code = NotFound desc = could not find container \"4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d\": container with ID starting with 4fd3567d3673e2beb4d1413b42ec6e94ff35719cb4e9a720ecec40a959a65e9d not found: ID does not exist" Jul 05 14:14:48.580769 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.580708 2569 scope.go:117] "RemoveContainer" containerID="e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb" Jul 05 14:14:48.580929 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:14:48.580912 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb\": container with ID starting with e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb not found: ID does not exist" containerID="e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb" Jul 05 14:14:48.580967 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.580938 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb"} err="failed to get container status \"e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb\": rpc error: code = NotFound desc = could not find container \"e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb\": container with ID starting with e76f4e3e42cf5c47e1ed3a274801c563ed08c3f1d9be4233c1f4e158056c37bb not found: ID does not exist" Jul 05 14:14:48.585687 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.585664 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh"] Jul 05 14:14:48.591510 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.591480 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-2d7b22-openshift-default-bbc7f5887-lpdqh"] Jul 05 14:14:48.798517 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:48.798490 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" path="/var/lib/kubelet/pods/855c91dc-f1ed-4e6d-87c5-e045936e5c3e/volumes" Jul 05 14:14:49.921865 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:49.921830 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh"] Jul 05 14:14:49.922228 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:49.922029 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" containerID="cri-o://fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64" gracePeriod=30 Jul 05 14:14:50.780778 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.780744 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2"] Jul 05 14:14:50.781123 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781101 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" Jul 05 14:14:50.781123 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781116 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" Jul 05 14:14:50.781123 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781128 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd2010f-f780-4575-99f9-70c2e2313773" containerName="main" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781134 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd2010f-f780-4575-99f9-70c2e2313773" containerName="main" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781157 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81d6088a-78e6-4fef-b1d0-9075c422be03" containerName="main" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781162 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="81d6088a-78e6-4fef-b1d0-9075c422be03" containerName="main" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781218 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781225 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781232 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="81d6088a-78e6-4fef-b1d0-9075c422be03" containerName="main" Jul 05 14:14:50.781398 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.781239 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd2010f-f780-4575-99f9-70c2e2313773" containerName="main" Jul 05 14:14:50.785653 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.785637 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.788739 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.788720 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-740454-openshift-default-dockercfg-zbkcp\"" Jul 05 14:14:50.803604 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.803580 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2"] Jul 05 14:14:50.843484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843455 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-token\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843594 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843498 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-data\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843594 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-certs\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843594 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843572 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-envoy\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843776 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8hd\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-kube-api-access-jk8hd\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843776 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843683 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-credential-socket\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843776 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843729 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7102d84-f4f2-4729-acaa-2224e6ed752a-istiod-ca-cert\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843776 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-socket\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.843959 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.843801 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-podinfo\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.944661 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944628 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-credential-socket\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7102d84-f4f2-4729-acaa-2224e6ed752a-istiod-ca-cert\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944698 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-socket\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-podinfo\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944811 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-token\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944844 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-data\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944874 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-certs\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944900 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-envoy\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945112 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.944924 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8hd\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-kube-api-access-jk8hd\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.945152 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-socket\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.945187 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-credential-socket\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.945335 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-data\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.945357 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-certs\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.945652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.945622 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7102d84-f4f2-4729-acaa-2224e6ed752a-istiod-ca-cert\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.947207 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.947186 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-envoy\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.947308 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.947297 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-podinfo\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.956802 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.956778 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8hd\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-kube-api-access-jk8hd\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:50.956916 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:50.956811 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-token\") pod \"e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:51.096940 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.096872 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:51.237492 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:14:51.237463 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7102d84_f4f2_4729_acaa_2224e6ed752a.slice/crio-d446709b007ac1ff9be9bd3520b04a8d298f59b85764762dd4ee81c7755008d3 WatchSource:0}: Error finding container d446709b007ac1ff9be9bd3520b04a8d298f59b85764762dd4ee81c7755008d3: Status 404 returned error can't find the container with id d446709b007ac1ff9be9bd3520b04a8d298f59b85764762dd4ee81c7755008d3 Jul 05 14:14:51.239612 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.239566 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:14:51.239736 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.239641 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:14:51.239736 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.239669 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:14:51.240785 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.240762 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2"] Jul 05 14:14:51.345822 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.345793 2569 patch_prober.go:28] interesting pod/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 05 14:14:51.345966 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.345850 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 05 14:14:51.580693 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.580659 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" event={"ID":"c7102d84-f4f2-4729-acaa-2224e6ed752a","Type":"ContainerStarted","Data":"504d3e2b6231ffdc80e76069c703ccf46d881ef5c20f19fc70ecda6bb555039a"} Jul 05 14:14:51.580884 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.580705 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" event={"ID":"c7102d84-f4f2-4729-acaa-2224e6ed752a","Type":"ContainerStarted","Data":"d446709b007ac1ff9be9bd3520b04a8d298f59b85764762dd4ee81c7755008d3"} Jul 05 14:14:51.607960 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:51.607902 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podStartSLOduration=1.6078871399999999 podStartE2EDuration="1.60788714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:14:51.604885478 +0000 UTC m=+2179.614355099" watchObservedRunningTime="2026-07-05 14:14:51.60788714 +0000 UTC m=+2179.617356760" Jul 05 14:14:52.097753 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:52.097715 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:52.102467 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:52.102440 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:52.585655 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:52.585626 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:52.586578 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:52.586559 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:14:55.188354 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.188294 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:14:55.285000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.284968 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-socket\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285193 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285023 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-data\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285193 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285068 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istiod-ca-cert\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285193 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285097 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-podinfo\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285193 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285172 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-envoy\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285446 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285202 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mtb4\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-kube-api-access-4mtb4\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285446 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285246 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:55.285446 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285341 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-certs\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285445 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-token\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285478 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-credential-socket\") pod \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\" (UID: \"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa\") " Jul 05 14:14:55.285722 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285599 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:55.285851 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285828 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:14:55.285944 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285866 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-data" (OuterVolumeSpecName: "istio-data") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:55.286000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285949 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.286000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285968 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.286000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285983 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.286000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.285994 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.286206 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.286038 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:55.287557 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.287525 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:14:55.287658 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.287580 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-kube-api-access-4mtb4" (OuterVolumeSpecName: "kube-api-access-4mtb4") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "kube-api-access-4mtb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:14:55.287787 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.287770 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-token" (OuterVolumeSpecName: "istio-token") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:14:55.288023 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.288008 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" (UID: "86465ce5-eb39-45fe-8fd3-6fea44ddbdfa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:14:55.386971 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.386942 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.386971 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.386966 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.386971 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.386975 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.387159 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.386984 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.387159 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.386992 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mtb4\" (UniqueName: \"kubernetes.io/projected/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa-kube-api-access-4mtb4\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:14:55.600376 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.600330 2569 generic.go:356] "Generic (PLEG): container finished" podID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerID="fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64" exitCode=0 Jul 05 14:14:55.600561 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.600395 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" Jul 05 14:14:55.600561 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.600414 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" event={"ID":"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa","Type":"ContainerDied","Data":"fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64"} Jul 05 14:14:55.600561 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.600495 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh" event={"ID":"86465ce5-eb39-45fe-8fd3-6fea44ddbdfa","Type":"ContainerDied","Data":"48fd44611c4ed29e05b7ef28bd4359831e5e9fa0e854ddc9610115c90e7cbf6a"} Jul 05 14:14:55.600561 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.600518 2569 scope.go:117] "RemoveContainer" containerID="fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64" Jul 05 14:14:55.609518 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.609500 2569 scope.go:117] "RemoveContainer" containerID="205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1" Jul 05 14:14:55.616859 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.616843 2569 scope.go:117] "RemoveContainer" containerID="fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64" Jul 05 14:14:55.617090 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:14:55.617074 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64\": container with ID starting with fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64 not found: ID does not exist" containerID="fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64" Jul 05 14:14:55.617126 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.617099 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64"} err="failed to get container status \"fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64\": rpc error: code = NotFound desc = could not find container \"fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64\": container with ID starting with fcfb4306973c6a035453c1b4cdeb63d05d20b1755e4b52c6011a439688e34d64 not found: ID does not exist" Jul 05 14:14:55.617126 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.617119 2569 scope.go:117] "RemoveContainer" containerID="205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1" Jul 05 14:14:55.617332 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:14:55.617317 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1\": container with ID starting with 205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1 not found: ID does not exist" containerID="205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1" Jul 05 14:14:55.617387 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.617337 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1"} err="failed to get container status \"205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1\": rpc error: code = NotFound desc = could not find container \"205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1\": container with ID starting with 205570130aef5060129f2972d12c37abb2f988a630692b5f8f935203044adaa1 not found: ID does not exist" Jul 05 14:14:55.623405 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.623384 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh"] Jul 05 14:14:55.629806 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:55.629786 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7c0371-openshift-default-5dc84c87b9-j6hsh"] Jul 05 14:14:56.805051 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:56.805019 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" path="/var/lib/kubelet/pods/86465ce5-eb39-45fe-8fd3-6fea44ddbdfa/volumes" Jul 05 14:14:58.911460 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911432 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x"] Jul 05 14:14:58.911820 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911782 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" Jul 05 14:14:58.911820 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911793 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" Jul 05 14:14:58.911820 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911812 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" Jul 05 14:14:58.911820 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911817 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" Jul 05 14:14:58.911963 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911829 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" Jul 05 14:14:58.911963 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911837 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="855c91dc-f1ed-4e6d-87c5-e045936e5c3e" containerName="istio-proxy" Jul 05 14:14:58.911963 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.911911 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" Jul 05 14:14:58.916312 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.916293 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:58.919413 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.919383 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-9394f3-openshift-default-dockercfg-72jfn\"" Jul 05 14:14:58.929083 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:58.929056 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x"] Jul 05 14:14:59.019425 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019387 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-certs\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019431 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-envoy\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019452 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-data\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019521 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-credential-socket\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019552 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d727a23-6d36-4818-aa3e-0473bb7da36f-istiod-ca-cert\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019603 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019594 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kcjn\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-kube-api-access-8kcjn\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019836 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019628 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-socket\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019836 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-podinfo\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.019836 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.019742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-token\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.120764 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.120731 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kcjn\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-kube-api-access-8kcjn\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.120945 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.120783 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-socket\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.120945 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.120806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-podinfo\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.120945 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.120848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-token\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121114 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121001 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-certs\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121114 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121042 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-envoy\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121114 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121078 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-data\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121257 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121120 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-credential-socket\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121257 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121146 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d727a23-6d36-4818-aa3e-0473bb7da36f-istiod-ca-cert\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121257 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121211 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-socket\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121441 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121327 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-certs\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121500 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121460 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-data\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121558 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121526 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-credential-socket\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.121889 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.121865 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d727a23-6d36-4818-aa3e-0473bb7da36f-istiod-ca-cert\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.123319 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.123300 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-envoy\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.123439 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.123327 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-podinfo\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.131882 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.131855 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-token\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.131986 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.131963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kcjn\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-kube-api-access-8kcjn\") pod \"e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.225621 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.225596 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:14:59.362339 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.362314 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x"] Jul 05 14:14:59.364294 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:14:59.364257 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d727a23_6d36_4818_aa3e_0473bb7da36f.slice/crio-2c7ec9df512c2482a2945499836b8cf67aec344e9114633a17046aa166c37b17 WatchSource:0}: Error finding container 2c7ec9df512c2482a2945499836b8cf67aec344e9114633a17046aa166c37b17: Status 404 returned error can't find the container with id 2c7ec9df512c2482a2945499836b8cf67aec344e9114633a17046aa166c37b17 Jul 05 14:14:59.367051 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.367014 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:14:59.367154 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.367086 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:14:59.367154 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.367119 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:14:59.623537 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.623450 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" event={"ID":"1d727a23-6d36-4818-aa3e-0473bb7da36f","Type":"ContainerStarted","Data":"1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633"} Jul 05 14:14:59.623537 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.623501 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" event={"ID":"1d727a23-6d36-4818-aa3e-0473bb7da36f","Type":"ContainerStarted","Data":"2c7ec9df512c2482a2945499836b8cf67aec344e9114633a17046aa166c37b17"} Jul 05 14:14:59.647518 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:14:59.647471 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podStartSLOduration=1.647454803 podStartE2EDuration="1.647454803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:14:59.64578992 +0000 UTC m=+2187.655259567" watchObservedRunningTime="2026-07-05 14:14:59.647454803 +0000 UTC m=+2187.656924422" Jul 05 14:15:00.226775 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:00.226746 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:00.231420 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:00.231391 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:00.628667 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:00.628587 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:00.629605 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:00.629582 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:08.627571 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:08.627521 2569 patch_prober.go:28] interesting pod/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 05 14:15:08.628069 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:08.627616 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 05 14:15:09.677250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:09.677165 2569 generic.go:356] "Generic (PLEG): container finished" podID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerID="504d3e2b6231ffdc80e76069c703ccf46d881ef5c20f19fc70ecda6bb555039a" exitCode=0 Jul 05 14:15:09.677250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:09.677239 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" event={"ID":"c7102d84-f4f2-4729-acaa-2224e6ed752a","Type":"ContainerDied","Data":"504d3e2b6231ffdc80e76069c703ccf46d881ef5c20f19fc70ecda6bb555039a"} Jul 05 14:15:09.677779 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:09.677678 2569 scope.go:117] "RemoveContainer" containerID="504d3e2b6231ffdc80e76069c703ccf46d881ef5c20f19fc70ecda6bb555039a" Jul 05 14:15:09.678662 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:09.678635 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:09.678743 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:09.678728 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:09.678781 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:09.678762 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:10.684059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:10.684026 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" event={"ID":"c7102d84-f4f2-4729-acaa-2224e6ed752a","Type":"ContainerStarted","Data":"99d52ac4d530bf57f3b92f2b9e6ae2f9679bbfa2c089d09150c7053b9a7cf574"} Jul 05 14:15:11.097828 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:11.097785 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:15:11.098156 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:11.098132 2569 patch_prober.go:28] interesting pod/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 05 14:15:11.098207 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:11.098187 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 05 14:15:12.097331 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:12.097289 2569 patch_prober.go:28] interesting pod/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 05 14:15:12.097851 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:12.097346 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 05 14:15:13.098058 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:13.098025 2569 patch_prober.go:28] interesting pod/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 05 14:15:13.098477 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:13.098080 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 05 14:15:14.097506 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:14.097462 2569 patch_prober.go:28] interesting pod/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Jul 05 14:15:14.097762 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:14.097520 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Jul 05 14:15:15.100993 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:15.100958 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:15:15.101510 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:15.101351 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:15:15.102091 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:15.102070 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:15:16.670530 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:16.670490 2569 patch_prober.go:28] interesting pod/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 05 14:15:16.670917 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:16.670548 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 05 14:15:17.720883 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:17.720848 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerID="1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633" exitCode=0 Jul 05 14:15:17.721392 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:17.720919 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" event={"ID":"1d727a23-6d36-4818-aa3e-0473bb7da36f","Type":"ContainerDied","Data":"1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633"} Jul 05 14:15:17.721392 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:17.721322 2569 scope.go:117] "RemoveContainer" containerID="1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633" Jul 05 14:15:17.722325 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:17.722288 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:17.722437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:17.722351 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:17.722437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:17.722404 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:18.736982 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:18.736936 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" event={"ID":"1d727a23-6d36-4818-aa3e-0473bb7da36f","Type":"ContainerStarted","Data":"1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d"} Jul 05 14:15:19.226091 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:19.226043 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:19.226344 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:19.226314 2569 patch_prober.go:28] interesting pod/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 05 14:15:19.226583 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:19.226555 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 05 14:15:20.225936 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:20.225891 2569 patch_prober.go:28] interesting pod/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 05 14:15:20.226441 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:20.225959 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 05 14:15:20.555240 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:20.555154 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2"] Jul 05 14:15:20.555450 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:20.555421 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" containerID="cri-o://99d52ac4d530bf57f3b92f2b9e6ae2f9679bbfa2c089d09150c7053b9a7cf574" gracePeriod=30 Jul 05 14:15:21.226392 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:21.226334 2569 patch_prober.go:28] interesting pod/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 05 14:15:21.226883 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:21.226418 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 05 14:15:22.226675 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:22.226222 2569 patch_prober.go:28] interesting pod/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 05 14:15:22.226675 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:22.226283 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 05 14:15:23.230073 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:23.230043 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:23.230484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:23.230297 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:23.230895 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:23.230877 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:25.775142 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.775109 2569 generic.go:356] "Generic (PLEG): container finished" podID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerID="99d52ac4d530bf57f3b92f2b9e6ae2f9679bbfa2c089d09150c7053b9a7cf574" exitCode=0 Jul 05 14:15:25.775513 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.775196 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" event={"ID":"c7102d84-f4f2-4729-acaa-2224e6ed752a","Type":"ContainerDied","Data":"99d52ac4d530bf57f3b92f2b9e6ae2f9679bbfa2c089d09150c7053b9a7cf574"} Jul 05 14:15:25.775513 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.775281 2569 scope.go:117] "RemoveContainer" containerID="504d3e2b6231ffdc80e76069c703ccf46d881ef5c20f19fc70ecda6bb555039a" Jul 05 14:15:25.806339 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.806318 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:15:25.891732 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.891657 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7102d84-f4f2-4729-acaa-2224e6ed752a-istiod-ca-cert\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.891732 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.891720 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-credential-socket\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.891916 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.891747 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jk8hd\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-kube-api-access-jk8hd\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.891967 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.891918 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-socket\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.892023 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.891966 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-token\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.892081 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892032 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-certs\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.892081 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892041 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:25.892081 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892064 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-data\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.892081 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892066 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7102d84-f4f2-4729-acaa-2224e6ed752a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:15:25.892282 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892090 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-podinfo\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.892282 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892127 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-envoy\") pod \"c7102d84-f4f2-4729-acaa-2224e6ed752a\" (UID: \"c7102d84-f4f2-4729-acaa-2224e6ed752a\") " Jul 05 14:15:25.892282 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892208 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:25.892448 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892285 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:25.892518 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892501 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.892571 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892521 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.892571 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892534 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7102d84-f4f2-4729-acaa-2224e6ed752a-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.892571 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.892547 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.893037 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.893007 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-data" (OuterVolumeSpecName: "istio-data") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:25.894025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.893999 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-kube-api-access-jk8hd" (OuterVolumeSpecName: "kube-api-access-jk8hd") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "kube-api-access-jk8hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:15:25.894226 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.894206 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:15:25.894599 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.894578 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:25.894599 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.894592 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-token" (OuterVolumeSpecName: "istio-token") pod "c7102d84-f4f2-4729-acaa-2224e6ed752a" (UID: "c7102d84-f4f2-4729-acaa-2224e6ed752a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:15:25.993484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.993458 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.993484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.993482 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.993628 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.993491 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.993628 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.993500 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jk8hd\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-kube-api-access-jk8hd\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:25.993628 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:25.993510 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7102d84-f4f2-4729-acaa-2224e6ed752a-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:26.781216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:26.781185 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" Jul 05 14:15:26.781216 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:26.781206 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2" event={"ID":"c7102d84-f4f2-4729-acaa-2224e6ed752a","Type":"ContainerDied","Data":"d446709b007ac1ff9be9bd3520b04a8d298f59b85764762dd4ee81c7755008d3"} Jul 05 14:15:26.781706 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:26.781268 2569 scope.go:117] "RemoveContainer" containerID="99d52ac4d530bf57f3b92f2b9e6ae2f9679bbfa2c089d09150c7053b9a7cf574" Jul 05 14:15:26.805665 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:26.805642 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2"] Jul 05 14:15:26.812197 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:26.812173 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-740454-openshift-default-56845dcbc9-v9zg2"] Jul 05 14:15:27.486436 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:27.486400 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x"] Jul 05 14:15:27.486654 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:27.486628 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" containerID="cri-o://1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d" gracePeriod=30 Jul 05 14:15:28.403413 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403358 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7"] Jul 05 14:15:28.403785 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403759 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" Jul 05 14:15:28.403785 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403772 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" Jul 05 14:15:28.403864 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403789 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" Jul 05 14:15:28.403864 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403795 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" Jul 05 14:15:28.403864 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403854 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="86465ce5-eb39-45fe-8fd3-6fea44ddbdfa" containerName="istio-proxy" Jul 05 14:15:28.403864 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.403864 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" Jul 05 14:15:28.406581 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.406563 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.410544 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.410523 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-9073be-openshift-default-dockercfg-5xxf2\"" Jul 05 14:15:28.428169 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.428145 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7"] Jul 05 14:15:28.515076 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515043 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-data\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515221 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515090 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-certs\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515221 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515144 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-envoy\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515221 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-podinfo\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515221 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccpqc\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-kube-api-access-ccpqc\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515221 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515213 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-token\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515447 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-credential-socket\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515447 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515296 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istiod-ca-cert\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.515447 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.515320 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-socket\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616273 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccpqc\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-kube-api-access-ccpqc\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616461 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-token\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616461 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616311 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-credential-socket\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616502 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istiod-ca-cert\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616547 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-socket\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616596 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-data\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616625 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-certs\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616668 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-envoy\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616831 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616678 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-credential-socket\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616831 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616686 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-podinfo\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616934 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616864 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-socket\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.616989 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.616953 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-data\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.617057 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.617020 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-certs\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.617313 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.617254 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istiod-ca-cert\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.618865 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.618845 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-envoy\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.618983 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.618954 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-podinfo\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.625687 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.625666 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-token\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.625907 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.625889 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccpqc\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-kube-api-access-ccpqc\") pod \"e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.717228 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.717197 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:28.800346 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.800301 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" path="/var/lib/kubelet/pods/c7102d84-f4f2-4729-acaa-2224e6ed752a/volumes" Jul 05 14:15:28.857683 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.857645 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7"] Jul 05 14:15:28.859887 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:15:28.859860 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3fdc9a7_189f_4baf_9383_395d4f8e1d6d.slice/crio-505b73a313876d9980e530a6e5f42c6a8e2fb302bb91884159df9aeeeea32c2e WatchSource:0}: Error finding container 505b73a313876d9980e530a6e5f42c6a8e2fb302bb91884159df9aeeeea32c2e: Status 404 returned error can't find the container with id 505b73a313876d9980e530a6e5f42c6a8e2fb302bb91884159df9aeeeea32c2e Jul 05 14:15:28.862016 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.861982 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:28.862072 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.862045 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:28.862109 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:28.862072 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:29.799416 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:29.799380 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" event={"ID":"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d","Type":"ContainerStarted","Data":"eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e"} Jul 05 14:15:29.799910 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:29.799426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" event={"ID":"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d","Type":"ContainerStarted","Data":"505b73a313876d9980e530a6e5f42c6a8e2fb302bb91884159df9aeeeea32c2e"} Jul 05 14:15:29.823132 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:29.823085 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" podStartSLOduration=1.823069787 podStartE2EDuration="1.823069787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:15:29.822426681 +0000 UTC m=+2217.831896321" watchObservedRunningTime="2026-07-05 14:15:29.823069787 +0000 UTC m=+2217.832539407" Jul 05 14:15:30.717537 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:30.717501 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:30.723459 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:30.723422 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:30.803447 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:30.803417 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:30.804765 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:30.804741 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:32.736543 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.736521 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:32.815011 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.814915 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerID="1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d" exitCode=0 Jul 05 14:15:32.815011 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.814972 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" Jul 05 14:15:32.815011 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.814995 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" event={"ID":"1d727a23-6d36-4818-aa3e-0473bb7da36f","Type":"ContainerDied","Data":"1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d"} Jul 05 14:15:32.815276 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.815052 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x" event={"ID":"1d727a23-6d36-4818-aa3e-0473bb7da36f","Type":"ContainerDied","Data":"2c7ec9df512c2482a2945499836b8cf67aec344e9114633a17046aa166c37b17"} Jul 05 14:15:32.815276 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.815077 2569 scope.go:117] "RemoveContainer" containerID="1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d" Jul 05 14:15:32.823635 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.823619 2569 scope.go:117] "RemoveContainer" containerID="1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633" Jul 05 14:15:32.830962 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.830948 2569 scope.go:117] "RemoveContainer" containerID="1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d" Jul 05 14:15:32.831209 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:15:32.831183 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d\": container with ID starting with 1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d not found: ID does not exist" containerID="1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d" Jul 05 14:15:32.831266 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.831210 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d"} err="failed to get container status \"1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d\": rpc error: code = NotFound desc = could not find container \"1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d\": container with ID starting with 1fb03e4e51801b3f9d8f381dda7329419305d68b48fe6411652f44d27d9bb12d not found: ID does not exist" Jul 05 14:15:32.831266 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.831226 2569 scope.go:117] "RemoveContainer" containerID="1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633" Jul 05 14:15:32.831477 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:15:32.831460 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633\": container with ID starting with 1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633 not found: ID does not exist" containerID="1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633" Jul 05 14:15:32.831530 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.831481 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633"} err="failed to get container status \"1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633\": rpc error: code = NotFound desc = could not find container \"1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633\": container with ID starting with 1d56f6ce99b6065a07c98a180217a31ca2be383eff97f246ee3058ca05279633 not found: ID does not exist" Jul 05 14:15:32.859800 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.859775 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-certs\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.859898 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.859814 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-token\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.859898 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.859833 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kcjn\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-kube-api-access-8kcjn\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.859990 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.859961 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-socket\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.860041 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860025 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-credential-socket\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.860084 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860068 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-data\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.860137 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860075 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:32.860137 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860122 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d727a23-6d36-4818-aa3e-0473bb7da36f-istiod-ca-cert\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.860245 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860154 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-podinfo\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.860245 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860185 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-envoy\") pod \"1d727a23-6d36-4818-aa3e-0473bb7da36f\" (UID: \"1d727a23-6d36-4818-aa3e-0473bb7da36f\") " Jul 05 14:15:32.860245 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860205 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:32.860414 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860334 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:32.860574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860551 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d727a23-6d36-4818-aa3e-0473bb7da36f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:15:32.860758 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860734 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.860840 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860760 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.860840 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860774 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d727a23-6d36-4818-aa3e-0473bb7da36f-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.860840 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860787 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.861001 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.860952 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-data" (OuterVolumeSpecName: "istio-data") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:32.862147 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.862124 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-token" (OuterVolumeSpecName: "istio-token") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:15:32.862250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.862154 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-kube-api-access-8kcjn" (OuterVolumeSpecName: "kube-api-access-8kcjn") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "kube-api-access-8kcjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:15:32.862355 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.862334 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:15:32.862615 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.862598 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1d727a23-6d36-4818-aa3e-0473bb7da36f" (UID: "1d727a23-6d36-4818-aa3e-0473bb7da36f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:15:32.961811 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.961790 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.961897 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.961818 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kcjn\" (UniqueName: \"kubernetes.io/projected/1d727a23-6d36-4818-aa3e-0473bb7da36f-kube-api-access-8kcjn\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.961897 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.961833 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.961897 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.961846 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:32.961897 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:32.961858 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d727a23-6d36-4818-aa3e-0473bb7da36f-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:15:33.148748 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:33.148724 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x"] Jul 05 14:15:33.157853 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:33.157832 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-9394f3-openshift-default-89cb95457-rtn4x"] Jul 05 14:15:34.798805 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:34.798769 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" path="/var/lib/kubelet/pods/1d727a23-6d36-4818-aa3e-0473bb7da36f/volumes" Jul 05 14:15:36.887738 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.887706 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457"] Jul 05 14:15:36.888156 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.888098 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" Jul 05 14:15:36.888156 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.888110 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" Jul 05 14:15:36.888242 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.888175 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7102d84-f4f2-4729-acaa-2224e6ed752a" containerName="istio-proxy" Jul 05 14:15:36.888242 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.888188 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" Jul 05 14:15:36.888242 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.888193 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" Jul 05 14:15:36.891139 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.891122 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.894096 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.894075 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-a7c515-openshift-default-dockercfg-2vdzc\"" Jul 05 14:15:36.904523 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.904505 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457"] Jul 05 14:15:36.995048 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995016 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istiod-ca-cert\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995217 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-podinfo\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995217 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995110 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-token\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995217 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-envoy\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995217 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995187 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-data\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995478 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995223 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-certs\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995478 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995295 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrwb\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-kube-api-access-bsrwb\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995478 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995349 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-socket\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:36.995478 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:36.995411 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-credential-socket\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096239 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-certs\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096431 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrwb\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-kube-api-access-bsrwb\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096431 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096354 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-socket\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096551 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-credential-socket\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096551 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istiod-ca-cert\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096649 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096551 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-podinfo\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096649 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096574 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-token\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096649 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096605 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-envoy\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096649 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-data\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096890 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096665 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-certs\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096890 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096756 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-socket\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.096890 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.096883 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-credential-socket\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.097104 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.097056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-data\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.097323 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.097288 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istiod-ca-cert\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.098900 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.098877 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-envoy\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.099273 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.099257 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-podinfo\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.105612 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.105592 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-token\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.105781 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.105763 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrwb\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-kube-api-access-bsrwb\") pod \"e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.201620 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.201587 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:37.381397 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.381357 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457"] Jul 05 14:15:37.383533 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:15:37.383504 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed16cfe_bbd2_4d0b_922a_c78f43fefb43.slice/crio-af4321cf652c6d3dc2df8f9c7a75676d736b5f1593a87f7441356a1e18bd8c2c WatchSource:0}: Error finding container af4321cf652c6d3dc2df8f9c7a75676d736b5f1593a87f7441356a1e18bd8c2c: Status 404 returned error can't find the container with id af4321cf652c6d3dc2df8f9c7a75676d736b5f1593a87f7441356a1e18bd8c2c Jul 05 14:15:37.385827 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.385796 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:37.385950 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.385854 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:37.385950 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.385884 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:37.846204 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.846106 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" event={"ID":"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43","Type":"ContainerStarted","Data":"b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc"} Jul 05 14:15:37.846204 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.846161 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" event={"ID":"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43","Type":"ContainerStarted","Data":"af4321cf652c6d3dc2df8f9c7a75676d736b5f1593a87f7441356a1e18bd8c2c"} Jul 05 14:15:37.879047 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:37.878997 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podStartSLOduration=1.8789792360000002 podStartE2EDuration="1.878979236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:15:37.877691663 +0000 UTC m=+2225.887161286" watchObservedRunningTime="2026-07-05 14:15:37.878979236 +0000 UTC m=+2225.888448858" Jul 05 14:15:38.201881 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:38.201852 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:38.203492 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:38.203466 2569 patch_prober.go:28] interesting pod/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 05 14:15:38.203607 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:38.203527 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 05 14:15:39.206664 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:39.206634 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:39.856692 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:39.856659 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:39.857584 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:39.857561 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:43.878176 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:43.878139 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerID="eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e" exitCode=0 Jul 05 14:15:43.878738 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:43.878215 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" event={"ID":"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d","Type":"ContainerDied","Data":"eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e"} Jul 05 14:15:43.878738 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:43.878684 2569 scope.go:117] "RemoveContainer" containerID="eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e" Jul 05 14:15:43.879673 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:43.879645 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:43.879762 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:43.879712 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:43.879762 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:43.879743 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:44.885345 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:44.885311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" event={"ID":"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d","Type":"ContainerStarted","Data":"0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4"} Jul 05 14:15:45.717817 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:45.717783 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:45.719176 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:45.719136 2569 patch_prober.go:28] interesting pod/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 05 14:15:45.719314 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:45.719213 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 05 14:15:45.803896 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:45.803868 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:46.718020 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:46.717988 2569 patch_prober.go:28] interesting pod/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 05 14:15:46.718513 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:46.718048 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 05 14:15:47.718409 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:47.718348 2569 patch_prober.go:28] interesting pod/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 05 14:15:47.718980 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:47.718947 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 05 14:15:48.718123 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:48.718078 2569 patch_prober.go:28] interesting pod/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Jul 05 14:15:48.718357 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:48.718140 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Jul 05 14:15:49.730831 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:49.730781 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:49.734934 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:49.734910 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:15:55.000448 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:54.999507 2569 patch_prober.go:28] interesting pod/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:58622->10.132.0.49:15021: read: connection reset by peer" start-of-body= Jul 05 14:15:55.000448 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:54.999574 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": read tcp 10.132.0.2:58622->10.132.0.49:15021: read: connection reset by peer" Jul 05 14:15:55.940169 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:55.940134 2569 generic.go:356] "Generic (PLEG): container finished" podID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerID="b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc" exitCode=0 Jul 05 14:15:55.940351 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:55.940207 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" event={"ID":"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43","Type":"ContainerDied","Data":"b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc"} Jul 05 14:15:55.940655 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:55.940639 2569 scope.go:117] "RemoveContainer" containerID="b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc" Jul 05 14:15:55.941633 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:55.941603 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:55.941694 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:55.941668 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:55.941729 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:55.941698 2569 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 05 14:15:56.949247 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:56.949213 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" event={"ID":"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43","Type":"ContainerStarted","Data":"6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133"} Jul 05 14:15:57.202296 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:57.202200 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:15:57.202475 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:57.202437 2569 patch_prober.go:28] interesting pod/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 05 14:15:57.202524 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:57.202483 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 05 14:15:58.202653 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:58.202622 2569 patch_prober.go:28] interesting pod/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 05 14:15:58.203064 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:58.202671 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 05 14:15:59.202584 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:59.202547 2569 patch_prober.go:28] interesting pod/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 05 14:15:59.202790 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:15:59.202619 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 05 14:16:00.202747 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.202713 2569 patch_prober.go:28] interesting pod/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 05 14:16:00.202916 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.202778 2569 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 05 14:16:00.279389 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.276775 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc"] Jul 05 14:16:00.279389 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.277692 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" Jul 05 14:16:00.279389 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.277714 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d727a23-6d36-4818-aa3e-0473bb7da36f" containerName="istio-proxy" Jul 05 14:16:00.282627 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.282588 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.286198 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.286161 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"kube-root-ca.crt\"" Jul 05 14:16:00.287666 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.287643 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"default-dockercfg-bmxrv\"" Jul 05 14:16:00.287779 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.287654 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"auth-test-model-9073be-kserve-self-signed-certs\"" Jul 05 14:16:00.287779 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.287715 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"openshift-service-ca.crt\"" Jul 05 14:16:00.290177 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.290155 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc"] Jul 05 14:16:00.344265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.343886 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ffqd\" (UniqueName: \"kubernetes.io/projected/07669747-92e7-49fc-b63d-4297b313b31b-kube-api-access-7ffqd\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.344265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.344018 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07669747-92e7-49fc-b63d-4297b313b31b-tls-certs\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.344265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.344061 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-model-cache\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.344265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.344086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-tmp-dir\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.344265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.344127 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-home\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.344265 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.344161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-dshm\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.444848 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.444815 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-home\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.444856 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-dshm\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.444910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ffqd\" (UniqueName: \"kubernetes.io/projected/07669747-92e7-49fc-b63d-4297b313b31b-kube-api-access-7ffqd\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.444975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07669747-92e7-49fc-b63d-4297b313b31b-tls-certs\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.444996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-model-cache\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.445019 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-tmp-dir\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445318 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.445275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-home\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445318 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.445297 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-model-cache\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.445490 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.445469 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-tmp-dir\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.447144 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.447121 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-dshm\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.447833 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.447815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07669747-92e7-49fc-b63d-4297b313b31b-tls-certs\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.453483 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.453436 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ffqd\" (UniqueName: \"kubernetes.io/projected/07669747-92e7-49fc-b63d-4297b313b31b-kube-api-access-7ffqd\") pod \"auth-test-model-9073be-kserve-77b57489f5-8hlhc\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.595079 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.595042 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:00.761386 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.761329 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc"] Jul 05 14:16:00.762695 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:16:00.762663 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07669747_92e7_49fc_b63d_4297b313b31b.slice/crio-62d8319c188b674dc5cd38f00954d8d1afb706daa4c92d2525e0a22b5c51b4e6 WatchSource:0}: Error finding container 62d8319c188b674dc5cd38f00954d8d1afb706daa4c92d2525e0a22b5c51b4e6: Status 404 returned error can't find the container with id 62d8319c188b674dc5cd38f00954d8d1afb706daa4c92d2525e0a22b5c51b4e6 Jul 05 14:16:00.971836 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.971802 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" event={"ID":"07669747-92e7-49fc-b63d-4297b313b31b","Type":"ContainerStarted","Data":"f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a"} Jul 05 14:16:00.971999 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.971849 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" event={"ID":"07669747-92e7-49fc-b63d-4297b313b31b","Type":"ContainerStarted","Data":"62d8319c188b674dc5cd38f00954d8d1afb706daa4c92d2525e0a22b5c51b4e6"} Jul 05 14:16:00.989970 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:00.989930 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" podStartSLOduration=0.989915681 podStartE2EDuration="989.915681ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:16:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:16:00.988541579 +0000 UTC m=+2248.998011201" watchObservedRunningTime="2026-07-05 14:16:00.989915681 +0000 UTC m=+2248.999385300" Jul 05 14:16:01.206526 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:01.206497 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:16:01.206896 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:01.206728 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:16:01.207432 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:01.207414 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:16:10.595997 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:10.595962 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:10.596431 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:10.596042 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:10.608725 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:10.608702 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:11.031107 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:11.031076 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:16:24.993784 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:24.993752 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc"] Jul 05 14:16:24.997312 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:24.997296 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.000428 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.000402 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"default-dockercfg-8v26t\"" Jul 05 14:16:25.001736 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.001719 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"openshift-service-ca.crt\"" Jul 05 14:16:25.001854 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.001827 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"auth-test-model-a7c515-kserve-self-signed-certs\"" Jul 05 14:16:25.001854 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.001836 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"kube-root-ca.crt\"" Jul 05 14:16:25.006778 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.006757 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc"] Jul 05 14:16:25.076454 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.076425 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-model-cache\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.076579 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.076472 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-dshm\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.076579 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.076565 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-home\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.076662 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.076592 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c757587d-459c-4469-8010-211be6a5416e-tls-certs\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.076662 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.076614 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29bj5\" (UniqueName: \"kubernetes.io/projected/c757587d-459c-4469-8010-211be6a5416e-kube-api-access-29bj5\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.076737 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.076685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-tmp-dir\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177416 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-dshm\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-home\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177481 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c757587d-459c-4469-8010-211be6a5416e-tls-certs\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177499 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29bj5\" (UniqueName: \"kubernetes.io/projected/c757587d-459c-4469-8010-211be6a5416e-kube-api-access-29bj5\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-tmp-dir\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177762 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177580 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-model-cache\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.177892 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177872 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-home\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.178000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177957 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-model-cache\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.178000 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.177987 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-tmp-dir\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.179615 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.179589 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-dshm\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.179994 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.179977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c757587d-459c-4469-8010-211be6a5416e-tls-certs\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.186437 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.186414 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29bj5\" (UniqueName: \"kubernetes.io/projected/c757587d-459c-4469-8010-211be6a5416e-kube-api-access-29bj5\") pod \"auth-test-model-a7c515-kserve-5cdcb58859-8cscc\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.308485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.308394 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:25.436951 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:25.436927 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc"] Jul 05 14:16:25.438859 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:16:25.438832 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc757587d_459c_4469_8010_211be6a5416e.slice/crio-bc079a9c415c590b8f8ff44f7c109db831558dad4551a6108ea7b6fd8c615ce8 WatchSource:0}: Error finding container bc079a9c415c590b8f8ff44f7c109db831558dad4551a6108ea7b6fd8c615ce8: Status 404 returned error can't find the container with id bc079a9c415c590b8f8ff44f7c109db831558dad4551a6108ea7b6fd8c615ce8 Jul 05 14:16:26.092092 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:26.092051 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" event={"ID":"c757587d-459c-4469-8010-211be6a5416e","Type":"ContainerStarted","Data":"c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53"} Jul 05 14:16:26.092517 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:26.092102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" event={"ID":"c757587d-459c-4469-8010-211be6a5416e","Type":"ContainerStarted","Data":"bc079a9c415c590b8f8ff44f7c109db831558dad4551a6108ea7b6fd8c615ce8"} Jul 05 14:16:26.111312 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:26.111265 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" podStartSLOduration=2.111250617 podStartE2EDuration="2.111250617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:16:26.109979647 +0000 UTC m=+2274.119449269" watchObservedRunningTime="2026-07-05 14:16:26.111250617 +0000 UTC m=+2274.120720237" Jul 05 14:16:35.309285 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:35.309251 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:35.309711 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:35.309300 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:35.321834 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:35.321811 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:16:36.150661 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:16:36.150635 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:18:12.378036 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:12.378000 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc"] Jul 05 14:18:13.402698 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:13.402664 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc"] Jul 05 14:18:27.397842 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.397784 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" podUID="07669747-92e7-49fc-b63d-4297b313b31b" containerName="main" containerID="cri-o://f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a" gracePeriod=45 Jul 05 14:18:27.645707 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.645684 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:18:27.685723 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.685647 2569 generic.go:356] "Generic (PLEG): container finished" podID="07669747-92e7-49fc-b63d-4297b313b31b" containerID="f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a" exitCode=0 Jul 05 14:18:27.685847 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.685735 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" Jul 05 14:18:27.685847 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.685776 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" event={"ID":"07669747-92e7-49fc-b63d-4297b313b31b","Type":"ContainerDied","Data":"f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a"} Jul 05 14:18:27.685847 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.685828 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc" event={"ID":"07669747-92e7-49fc-b63d-4297b313b31b","Type":"ContainerDied","Data":"62d8319c188b674dc5cd38f00954d8d1afb706daa4c92d2525e0a22b5c51b4e6"} Jul 05 14:18:27.685963 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.685853 2569 scope.go:117] "RemoveContainer" containerID="f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a" Jul 05 14:18:27.695041 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.695014 2569 scope.go:117] "RemoveContainer" containerID="f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a" Jul 05 14:18:27.695295 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:18:27.695277 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a\": container with ID starting with f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a not found: ID does not exist" containerID="f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a" Jul 05 14:18:27.695360 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.695310 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a"} err="failed to get container status \"f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a\": rpc error: code = NotFound desc = could not find container \"f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a\": container with ID starting with f50906a91d0c8ce87ffd8e8584b8d03c1d0a6c6163823d7b946732e4e0d9105a not found: ID does not exist" Jul 05 14:18:27.796029 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.795992 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-model-cache\") pod \"07669747-92e7-49fc-b63d-4297b313b31b\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " Jul 05 14:18:27.796208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796051 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07669747-92e7-49fc-b63d-4297b313b31b-tls-certs\") pod \"07669747-92e7-49fc-b63d-4297b313b31b\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " Jul 05 14:18:27.796208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796105 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-dshm\") pod \"07669747-92e7-49fc-b63d-4297b313b31b\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " Jul 05 14:18:27.796208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796159 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ffqd\" (UniqueName: \"kubernetes.io/projected/07669747-92e7-49fc-b63d-4297b313b31b-kube-api-access-7ffqd\") pod \"07669747-92e7-49fc-b63d-4297b313b31b\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " Jul 05 14:18:27.796208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796186 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-tmp-dir\") pod \"07669747-92e7-49fc-b63d-4297b313b31b\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " Jul 05 14:18:27.796456 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796209 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-home\") pod \"07669747-92e7-49fc-b63d-4297b313b31b\" (UID: \"07669747-92e7-49fc-b63d-4297b313b31b\") " Jul 05 14:18:27.796456 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796343 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-model-cache" (OuterVolumeSpecName: "model-cache") pod "07669747-92e7-49fc-b63d-4297b313b31b" (UID: "07669747-92e7-49fc-b63d-4297b313b31b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:27.796561 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796493 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "07669747-92e7-49fc-b63d-4297b313b31b" (UID: "07669747-92e7-49fc-b63d-4297b313b31b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:27.796561 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796513 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-home" (OuterVolumeSpecName: "home") pod "07669747-92e7-49fc-b63d-4297b313b31b" (UID: "07669747-92e7-49fc-b63d-4297b313b31b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:27.796642 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.796575 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:27.798472 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.798449 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07669747-92e7-49fc-b63d-4297b313b31b-kube-api-access-7ffqd" (OuterVolumeSpecName: "kube-api-access-7ffqd") pod "07669747-92e7-49fc-b63d-4297b313b31b" (UID: "07669747-92e7-49fc-b63d-4297b313b31b"). InnerVolumeSpecName "kube-api-access-7ffqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:18:27.798576 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.798490 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07669747-92e7-49fc-b63d-4297b313b31b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "07669747-92e7-49fc-b63d-4297b313b31b" (UID: "07669747-92e7-49fc-b63d-4297b313b31b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:18:27.798576 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.798532 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-dshm" (OuterVolumeSpecName: "dshm") pod "07669747-92e7-49fc-b63d-4297b313b31b" (UID: "07669747-92e7-49fc-b63d-4297b313b31b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:27.897032 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.897008 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:27.897032 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.897031 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ffqd\" (UniqueName: \"kubernetes.io/projected/07669747-92e7-49fc-b63d-4297b313b31b-kube-api-access-7ffqd\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:27.897171 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.897041 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:27.897171 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.897049 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07669747-92e7-49fc-b63d-4297b313b31b-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:27.897171 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:27.897056 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07669747-92e7-49fc-b63d-4297b313b31b-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:28.008721 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.008690 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc"] Jul 05 14:18:28.012818 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.012793 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/auth-test-model-9073be-kserve-77b57489f5-8hlhc"] Jul 05 14:18:28.423484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.423386 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" podUID="c757587d-459c-4469-8010-211be6a5416e" containerName="main" containerID="cri-o://c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53" gracePeriod=45 Jul 05 14:18:28.673853 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.673802 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:18:28.693059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.693027 2569 generic.go:356] "Generic (PLEG): container finished" podID="c757587d-459c-4469-8010-211be6a5416e" containerID="c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53" exitCode=0 Jul 05 14:18:28.693173 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.693079 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" Jul 05 14:18:28.693173 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.693103 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" event={"ID":"c757587d-459c-4469-8010-211be6a5416e","Type":"ContainerDied","Data":"c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53"} Jul 05 14:18:28.693173 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.693131 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc" event={"ID":"c757587d-459c-4469-8010-211be6a5416e","Type":"ContainerDied","Data":"bc079a9c415c590b8f8ff44f7c109db831558dad4551a6108ea7b6fd8c615ce8"} Jul 05 14:18:28.693173 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.693148 2569 scope.go:117] "RemoveContainer" containerID="c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53" Jul 05 14:18:28.702848 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.702831 2569 scope.go:117] "RemoveContainer" containerID="c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53" Jul 05 14:18:28.703097 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:18:28.703070 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53\": container with ID starting with c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53 not found: ID does not exist" containerID="c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53" Jul 05 14:18:28.703193 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.703104 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53"} err="failed to get container status \"c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53\": rpc error: code = NotFound desc = could not find container \"c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53\": container with ID starting with c3b72c72d91c6f52a10df4a35ca995fb6d26042336f4f46a252098488d8abc53 not found: ID does not exist" Jul 05 14:18:28.799613 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.799584 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07669747-92e7-49fc-b63d-4297b313b31b" path="/var/lib/kubelet/pods/07669747-92e7-49fc-b63d-4297b313b31b/volumes" Jul 05 14:18:28.804086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804067 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-model-cache\") pod \"c757587d-459c-4469-8010-211be6a5416e\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " Jul 05 14:18:28.804208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804091 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-home\") pod \"c757587d-459c-4469-8010-211be6a5416e\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " Jul 05 14:18:28.804208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804173 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c757587d-459c-4469-8010-211be6a5416e-tls-certs\") pod \"c757587d-459c-4469-8010-211be6a5416e\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " Jul 05 14:18:28.804208 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804199 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-tmp-dir\") pod \"c757587d-459c-4469-8010-211be6a5416e\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " Jul 05 14:18:28.804319 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804262 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29bj5\" (UniqueName: \"kubernetes.io/projected/c757587d-459c-4469-8010-211be6a5416e-kube-api-access-29bj5\") pod \"c757587d-459c-4469-8010-211be6a5416e\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " Jul 05 14:18:28.804319 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804299 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-dshm\") pod \"c757587d-459c-4469-8010-211be6a5416e\" (UID: \"c757587d-459c-4469-8010-211be6a5416e\") " Jul 05 14:18:28.804433 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804327 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-model-cache" (OuterVolumeSpecName: "model-cache") pod "c757587d-459c-4469-8010-211be6a5416e" (UID: "c757587d-459c-4469-8010-211be6a5416e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:28.804497 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804445 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-home" (OuterVolumeSpecName: "home") pod "c757587d-459c-4469-8010-211be6a5416e" (UID: "c757587d-459c-4469-8010-211be6a5416e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:28.804555 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804537 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c757587d-459c-4469-8010-211be6a5416e" (UID: "c757587d-459c-4469-8010-211be6a5416e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:28.804647 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804627 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:28.804742 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804652 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:28.804742 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.804670 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:28.806356 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.806333 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c757587d-459c-4469-8010-211be6a5416e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c757587d-459c-4469-8010-211be6a5416e" (UID: "c757587d-459c-4469-8010-211be6a5416e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:18:28.806512 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.806493 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-dshm" (OuterVolumeSpecName: "dshm") pod "c757587d-459c-4469-8010-211be6a5416e" (UID: "c757587d-459c-4469-8010-211be6a5416e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:18:28.806560 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.806521 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c757587d-459c-4469-8010-211be6a5416e-kube-api-access-29bj5" (OuterVolumeSpecName: "kube-api-access-29bj5") pod "c757587d-459c-4469-8010-211be6a5416e" (UID: "c757587d-459c-4469-8010-211be6a5416e"). InnerVolumeSpecName "kube-api-access-29bj5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:18:28.905702 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.905670 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c757587d-459c-4469-8010-211be6a5416e-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:28.905702 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.905699 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29bj5\" (UniqueName: \"kubernetes.io/projected/c757587d-459c-4469-8010-211be6a5416e-kube-api-access-29bj5\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:28.905862 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:28.905714 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c757587d-459c-4469-8010-211be6a5416e-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:18:29.016294 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:29.016266 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc"] Jul 05 14:18:29.019950 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:29.019919 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/auth-test-model-a7c515-kserve-5cdcb58859-8cscc"] Jul 05 14:18:30.799610 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:30.799577 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c757587d-459c-4469-8010-211be6a5416e" path="/var/lib/kubelet/pods/c757587d-459c-4469-8010-211be6a5416e/volumes" Jul 05 14:18:37.101884 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.101855 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g"] Jul 05 14:18:37.102254 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.102203 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07669747-92e7-49fc-b63d-4297b313b31b" containerName="main" Jul 05 14:18:37.102254 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.102214 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="07669747-92e7-49fc-b63d-4297b313b31b" containerName="main" Jul 05 14:18:37.102391 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.102257 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c757587d-459c-4469-8010-211be6a5416e" containerName="main" Jul 05 14:18:37.102391 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.102267 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c757587d-459c-4469-8010-211be6a5416e" containerName="main" Jul 05 14:18:37.102391 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.102357 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c757587d-459c-4469-8010-211be6a5416e" containerName="main" Jul 05 14:18:37.102391 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.102389 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="07669747-92e7-49fc-b63d-4297b313b31b" containerName="main" Jul 05 14:18:37.106826 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.106806 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.109890 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.109871 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"kube-root-ca.crt\"" Jul 05 14:18:37.109991 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.109873 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"openshift-service-ca.crt\"" Jul 05 14:18:37.111164 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.111145 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"default-dockercfg-bmxrv\"" Jul 05 14:18:37.111276 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.111206 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-9073be\"/\"sub-test-model-9073be-kserve-self-signed-certs\"" Jul 05 14:18:37.115155 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.115132 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g"] Jul 05 14:18:37.279848 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.279815 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-model-cache\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.280013 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.279856 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tmp-dir\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.280013 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.279888 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tls-certs\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.280092 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.280004 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vt9w\" (UniqueName: \"kubernetes.io/projected/6e858b87-2c01-45a1-a0a0-14d7598bcbad-kube-api-access-2vt9w\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.280092 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.280084 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-dshm\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.280158 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.280106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-home\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.381728 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.381620 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vt9w\" (UniqueName: \"kubernetes.io/projected/6e858b87-2c01-45a1-a0a0-14d7598bcbad-kube-api-access-2vt9w\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.381728 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.381716 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-dshm\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.381965 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.381745 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-home\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.381965 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.381799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-model-cache\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.381965 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.381837 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tmp-dir\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.381965 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.381872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tls-certs\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.382387 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.382340 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-model-cache\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.382745 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.382721 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-home\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.382952 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.382746 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tmp-dir\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.384690 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.384671 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tls-certs\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.385648 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.385626 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-dshm\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.390824 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.390805 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vt9w\" (UniqueName: \"kubernetes.io/projected/6e858b87-2c01-45a1-a0a0-14d7598bcbad-kube-api-access-2vt9w\") pod \"sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.416075 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.416052 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:37.554269 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.554246 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g"] Jul 05 14:18:37.556346 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:18:37.556322 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e858b87_2c01_45a1_a0a0_14d7598bcbad.slice/crio-3e3779a635b4357c53de3891388550090244d716026cf92f9c9cd127299a8206 WatchSource:0}: Error finding container 3e3779a635b4357c53de3891388550090244d716026cf92f9c9cd127299a8206: Status 404 returned error can't find the container with id 3e3779a635b4357c53de3891388550090244d716026cf92f9c9cd127299a8206 Jul 05 14:18:37.739944 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.739899 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" event={"ID":"6e858b87-2c01-45a1-a0a0-14d7598bcbad","Type":"ContainerStarted","Data":"52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14"} Jul 05 14:18:37.740113 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.739958 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" event={"ID":"6e858b87-2c01-45a1-a0a0-14d7598bcbad","Type":"ContainerStarted","Data":"3e3779a635b4357c53de3891388550090244d716026cf92f9c9cd127299a8206"} Jul 05 14:18:37.759964 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:37.759912 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" podStartSLOduration=0.759897084 podStartE2EDuration="759.897084ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:18:37.758277519 +0000 UTC m=+2405.767747141" watchObservedRunningTime="2026-07-05 14:18:37.759897084 +0000 UTC m=+2405.769366703" Jul 05 14:18:47.416203 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:47.416162 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:47.416203 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:47.416213 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:47.428726 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:47.428699 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:18:47.798806 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:18:47.798776 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:19:32.690960 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.690924 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg"] Jul 05 14:19:32.694725 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.694704 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.697558 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.697538 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"kube-root-ca.crt\"" Jul 05 14:19:32.697671 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.697616 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"openshift-service-ca.crt\"" Jul 05 14:19:32.698979 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.698958 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"default-dockercfg-8v26t\"" Jul 05 14:19:32.699061 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.699007 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"sub-test-model-a7c515-kserve-self-signed-certs\"" Jul 05 14:19:32.704043 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.704018 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg"] Jul 05 14:19:32.791988 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.791946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-dshm\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.792195 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.791997 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tmp-dir\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.792195 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.792026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tls-certs\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.792195 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.792086 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-home\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.792195 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.792142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwbb8\" (UniqueName: \"kubernetes.io/projected/b8c97b91-d83a-4103-b06b-bdcc2dacd680-kube-api-access-hwbb8\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.792195 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.792192 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-model-cache\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.893706 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.893670 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-home\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.893913 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.893721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwbb8\" (UniqueName: \"kubernetes.io/projected/b8c97b91-d83a-4103-b06b-bdcc2dacd680-kube-api-access-hwbb8\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.893913 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.893776 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-model-cache\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.893913 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.893829 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-dshm\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.894093 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.893917 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tmp-dir\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.894093 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.893958 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tls-certs\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.894205 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.894143 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-home\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.894247 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.894226 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-model-cache\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.894282 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.894271 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tmp-dir\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.896058 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.896024 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-dshm\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.896314 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.896295 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tls-certs\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:32.901931 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:32.901909 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwbb8\" (UniqueName: \"kubernetes.io/projected/b8c97b91-d83a-4103-b06b-bdcc2dacd680-kube-api-access-hwbb8\") pod \"sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:33.004866 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:33.004839 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:33.140816 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:33.140791 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg"] Jul 05 14:19:33.142735 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:19:33.142708 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8c97b91_d83a_4103_b06b_bdcc2dacd680.slice/crio-902c54ef51e6d73280090432bd58c1c201482a045dc9df30d00767d04f69c521 WatchSource:0}: Error finding container 902c54ef51e6d73280090432bd58c1c201482a045dc9df30d00767d04f69c521: Status 404 returned error can't find the container with id 902c54ef51e6d73280090432bd58c1c201482a045dc9df30d00767d04f69c521 Jul 05 14:19:34.011192 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:34.011154 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" event={"ID":"b8c97b91-d83a-4103-b06b-bdcc2dacd680","Type":"ContainerStarted","Data":"ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a"} Jul 05 14:19:34.011192 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:34.011194 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" event={"ID":"b8c97b91-d83a-4103-b06b-bdcc2dacd680","Type":"ContainerStarted","Data":"902c54ef51e6d73280090432bd58c1c201482a045dc9df30d00767d04f69c521"} Jul 05 14:19:34.031848 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:34.031803 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" podStartSLOduration=2.031789006 podStartE2EDuration="2.031789006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:19:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:19:34.030581365 +0000 UTC m=+2462.040050987" watchObservedRunningTime="2026-07-05 14:19:34.031789006 +0000 UTC m=+2462.041258625" Jul 05 14:19:43.005490 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:43.005443 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:43.005917 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:43.005510 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:43.017851 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:43.017826 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:19:43.068639 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:19:43.068616 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:20:26.367400 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:26.367303 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g"] Jul 05 14:20:27.237233 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:27.237196 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg"] Jul 05 14:20:41.386917 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.386860 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" podUID="6e858b87-2c01-45a1-a0a0-14d7598bcbad" containerName="main" containerID="cri-o://52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14" gracePeriod=45 Jul 05 14:20:41.638698 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.638634 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:20:41.824481 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824450 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tmp-dir\") pod \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " Jul 05 14:20:41.824481 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824489 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-home\") pod \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " Jul 05 14:20:41.824718 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824530 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vt9w\" (UniqueName: \"kubernetes.io/projected/6e858b87-2c01-45a1-a0a0-14d7598bcbad-kube-api-access-2vt9w\") pod \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " Jul 05 14:20:41.824718 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824556 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-dshm\") pod \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " Jul 05 14:20:41.824718 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824610 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tls-certs\") pod \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " Jul 05 14:20:41.824718 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824639 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-model-cache\") pod \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\" (UID: \"6e858b87-2c01-45a1-a0a0-14d7598bcbad\") " Jul 05 14:20:41.824931 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824747 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6e858b87-2c01-45a1-a0a0-14d7598bcbad" (UID: "6e858b87-2c01-45a1-a0a0-14d7598bcbad"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:41.824931 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824759 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-home" (OuterVolumeSpecName: "home") pod "6e858b87-2c01-45a1-a0a0-14d7598bcbad" (UID: "6e858b87-2c01-45a1-a0a0-14d7598bcbad"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:41.825044 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824965 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:41.825044 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824983 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-model-cache" (OuterVolumeSpecName: "model-cache") pod "6e858b87-2c01-45a1-a0a0-14d7598bcbad" (UID: "6e858b87-2c01-45a1-a0a0-14d7598bcbad"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:41.825044 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.824990 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:41.826729 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.826697 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-dshm" (OuterVolumeSpecName: "dshm") pod "6e858b87-2c01-45a1-a0a0-14d7598bcbad" (UID: "6e858b87-2c01-45a1-a0a0-14d7598bcbad"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:41.826841 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.826771 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6e858b87-2c01-45a1-a0a0-14d7598bcbad" (UID: "6e858b87-2c01-45a1-a0a0-14d7598bcbad"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:20:41.826841 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.826789 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e858b87-2c01-45a1-a0a0-14d7598bcbad-kube-api-access-2vt9w" (OuterVolumeSpecName: "kube-api-access-2vt9w") pod "6e858b87-2c01-45a1-a0a0-14d7598bcbad" (UID: "6e858b87-2c01-45a1-a0a0-14d7598bcbad"). InnerVolumeSpecName "kube-api-access-2vt9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:20:41.926226 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.926153 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vt9w\" (UniqueName: \"kubernetes.io/projected/6e858b87-2c01-45a1-a0a0-14d7598bcbad-kube-api-access-2vt9w\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:41.926226 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.926182 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:41.926226 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.926206 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e858b87-2c01-45a1-a0a0-14d7598bcbad-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:41.926226 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:41.926219 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e858b87-2c01-45a1-a0a0-14d7598bcbad-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.257123 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.257072 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" podUID="b8c97b91-d83a-4103-b06b-bdcc2dacd680" containerName="main" containerID="cri-o://ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a" gracePeriod=45 Jul 05 14:20:42.346533 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.346502 2569 generic.go:356] "Generic (PLEG): container finished" podID="6e858b87-2c01-45a1-a0a0-14d7598bcbad" containerID="52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14" exitCode=0 Jul 05 14:20:42.346660 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.346543 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" Jul 05 14:20:42.346660 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.346591 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" event={"ID":"6e858b87-2c01-45a1-a0a0-14d7598bcbad","Type":"ContainerDied","Data":"52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14"} Jul 05 14:20:42.346660 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.346639 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g" event={"ID":"6e858b87-2c01-45a1-a0a0-14d7598bcbad","Type":"ContainerDied","Data":"3e3779a635b4357c53de3891388550090244d716026cf92f9c9cd127299a8206"} Jul 05 14:20:42.346835 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.346661 2569 scope.go:117] "RemoveContainer" containerID="52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14" Jul 05 14:20:42.367958 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.367927 2569 scope.go:117] "RemoveContainer" containerID="52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14" Jul 05 14:20:42.368151 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.368136 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g"] Jul 05 14:20:42.368230 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:20:42.368214 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14\": container with ID starting with 52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14 not found: ID does not exist" containerID="52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14" Jul 05 14:20:42.368277 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.368234 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14"} err="failed to get container status \"52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14\": rpc error: code = NotFound desc = could not find container \"52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14\": container with ID starting with 52ea312064c01ed8a0cfbd2ea7f53437e3af4b5ce36c8fc39b61cf863b7ffc14 not found: ID does not exist" Jul 05 14:20:42.372514 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.372491 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-9073be/sub-test-model-9073be-kserve-84d4c5d4c5-tdh2g"] Jul 05 14:20:42.501302 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.501279 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:20:42.633686 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633587 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-home\") pod \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " Jul 05 14:20:42.633686 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633641 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwbb8\" (UniqueName: \"kubernetes.io/projected/b8c97b91-d83a-4103-b06b-bdcc2dacd680-kube-api-access-hwbb8\") pod \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " Jul 05 14:20:42.633686 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633662 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tmp-dir\") pod \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " Jul 05 14:20:42.633686 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633693 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tls-certs\") pod \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " Jul 05 14:20:42.634010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633711 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-dshm\") pod \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " Jul 05 14:20:42.634010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633757 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-model-cache\") pod \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\" (UID: \"b8c97b91-d83a-4103-b06b-bdcc2dacd680\") " Jul 05 14:20:42.634010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633896 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-home" (OuterVolumeSpecName: "home") pod "b8c97b91-d83a-4103-b06b-bdcc2dacd680" (UID: "b8c97b91-d83a-4103-b06b-bdcc2dacd680"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:42.634010 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.633971 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b8c97b91-d83a-4103-b06b-bdcc2dacd680" (UID: "b8c97b91-d83a-4103-b06b-bdcc2dacd680"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:42.634213 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.634052 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-model-cache" (OuterVolumeSpecName: "model-cache") pod "b8c97b91-d83a-4103-b06b-bdcc2dacd680" (UID: "b8c97b91-d83a-4103-b06b-bdcc2dacd680"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:42.634213 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.634127 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.634213 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.634141 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.634213 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.634151 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.635845 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.635824 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c97b91-d83a-4103-b06b-bdcc2dacd680-kube-api-access-hwbb8" (OuterVolumeSpecName: "kube-api-access-hwbb8") pod "b8c97b91-d83a-4103-b06b-bdcc2dacd680" (UID: "b8c97b91-d83a-4103-b06b-bdcc2dacd680"). InnerVolumeSpecName "kube-api-access-hwbb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:20:42.635845 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.635826 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b8c97b91-d83a-4103-b06b-bdcc2dacd680" (UID: "b8c97b91-d83a-4103-b06b-bdcc2dacd680"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:20:42.635993 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.635826 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-dshm" (OuterVolumeSpecName: "dshm") pod "b8c97b91-d83a-4103-b06b-bdcc2dacd680" (UID: "b8c97b91-d83a-4103-b06b-bdcc2dacd680"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:20:42.734745 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.734710 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwbb8\" (UniqueName: \"kubernetes.io/projected/b8c97b91-d83a-4103-b06b-bdcc2dacd680-kube-api-access-hwbb8\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.734745 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.734739 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c97b91-d83a-4103-b06b-bdcc2dacd680-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.734745 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.734748 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c97b91-d83a-4103-b06b-bdcc2dacd680-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:20:42.799513 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:42.799476 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e858b87-2c01-45a1-a0a0-14d7598bcbad" path="/var/lib/kubelet/pods/6e858b87-2c01-45a1-a0a0-14d7598bcbad/volumes" Jul 05 14:20:43.353080 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.353047 2569 generic.go:356] "Generic (PLEG): container finished" podID="b8c97b91-d83a-4103-b06b-bdcc2dacd680" containerID="ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a" exitCode=0 Jul 05 14:20:43.353250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.353086 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" Jul 05 14:20:43.353250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.353127 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" event={"ID":"b8c97b91-d83a-4103-b06b-bdcc2dacd680","Type":"ContainerDied","Data":"ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a"} Jul 05 14:20:43.353250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.353171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg" event={"ID":"b8c97b91-d83a-4103-b06b-bdcc2dacd680","Type":"ContainerDied","Data":"902c54ef51e6d73280090432bd58c1c201482a045dc9df30d00767d04f69c521"} Jul 05 14:20:43.353250 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.353189 2569 scope.go:117] "RemoveContainer" containerID="ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a" Jul 05 14:20:43.362424 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.362412 2569 scope.go:117] "RemoveContainer" containerID="ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a" Jul 05 14:20:43.362666 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:20:43.362639 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a\": container with ID starting with ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a not found: ID does not exist" containerID="ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a" Jul 05 14:20:43.362768 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.362674 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a"} err="failed to get container status \"ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a\": rpc error: code = NotFound desc = could not find container \"ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a\": container with ID starting with ae55b783e469bfc93a65912d8a315a2a72239b313c83c737524dd84dbfa29e1a not found: ID does not exist" Jul 05 14:20:43.370769 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.370745 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg"] Jul 05 14:20:43.375087 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:43.375064 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/sub-test-model-a7c515-kserve-55dd9f9ddc-bgdkg"] Jul 05 14:20:44.799503 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:20:44.799471 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8c97b91-d83a-4103-b06b-bdcc2dacd680" path="/var/lib/kubelet/pods/b8c97b91-d83a-4103-b06b-bdcc2dacd680/volumes" Jul 05 14:21:32.994862 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.994830 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27"] Jul 05 14:21:32.995290 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.995196 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e858b87-2c01-45a1-a0a0-14d7598bcbad" containerName="main" Jul 05 14:21:32.995290 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.995208 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e858b87-2c01-45a1-a0a0-14d7598bcbad" containerName="main" Jul 05 14:21:32.995290 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.995223 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8c97b91-d83a-4103-b06b-bdcc2dacd680" containerName="main" Jul 05 14:21:32.995290 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.995228 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8c97b91-d83a-4103-b06b-bdcc2dacd680" containerName="main" Jul 05 14:21:32.995290 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.995283 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8c97b91-d83a-4103-b06b-bdcc2dacd680" containerName="main" Jul 05 14:21:32.995475 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.995295 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e858b87-2c01-45a1-a0a0-14d7598bcbad" containerName="main" Jul 05 14:21:32.998270 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:32.998254 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.001331 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.001309 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"openshift-service-ca.crt\"" Jul 05 14:21:33.002594 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.002571 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"rate-test-model-a7c515-kserve-self-signed-certs\"" Jul 05 14:21:33.002594 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.002575 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"kube-root-ca.crt\"" Jul 05 14:21:33.002814 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.002793 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-a7c515\"/\"default-dockercfg-8v26t\"" Jul 05 14:21:33.007949 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.007918 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27"] Jul 05 14:21:33.082659 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.082632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-model-cache\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.082784 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.082690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-dshm\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.082784 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.082751 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6988cb-5b54-4d14-9339-b29a9d230aac-tls-certs\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.083054 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.082794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-tmp-dir\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.083054 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.082830 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xllqj\" (UniqueName: \"kubernetes.io/projected/2a6988cb-5b54-4d14-9339-b29a9d230aac-kube-api-access-xllqj\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.083054 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.082858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-home\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.183839 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.183806 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-model-cache\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184022 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.183866 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-dshm\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184022 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.183944 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6988cb-5b54-4d14-9339-b29a9d230aac-tls-certs\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184022 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.183990 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-tmp-dir\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184182 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.184028 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xllqj\" (UniqueName: \"kubernetes.io/projected/2a6988cb-5b54-4d14-9339-b29a9d230aac-kube-api-access-xllqj\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184182 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.184058 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-home\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184311 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.184280 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-model-cache\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184400 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.184345 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-tmp-dir\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.184473 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.184425 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-home\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.186309 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.186286 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-dshm\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.186409 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.186333 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6988cb-5b54-4d14-9339-b29a9d230aac-tls-certs\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.192695 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.192677 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xllqj\" (UniqueName: \"kubernetes.io/projected/2a6988cb-5b54-4d14-9339-b29a9d230aac-kube-api-access-xllqj\") pod \"rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.308030 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.307936 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:33.447416 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.447391 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27"] Jul 05 14:21:33.449149 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:21:33.449125 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a6988cb_5b54_4d14_9339_b29a9d230aac.slice/crio-43cc336d141665ba02d9d883084f0207117b3acdf2580bed8792bacbaf669b28 WatchSource:0}: Error finding container 43cc336d141665ba02d9d883084f0207117b3acdf2580bed8792bacbaf669b28: Status 404 returned error can't find the container with id 43cc336d141665ba02d9d883084f0207117b3acdf2580bed8792bacbaf669b28 Jul 05 14:21:33.608949 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.608813 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" event={"ID":"2a6988cb-5b54-4d14-9339-b29a9d230aac","Type":"ContainerStarted","Data":"e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1"} Jul 05 14:21:33.608949 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.608864 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" event={"ID":"2a6988cb-5b54-4d14-9339-b29a9d230aac","Type":"ContainerStarted","Data":"43cc336d141665ba02d9d883084f0207117b3acdf2580bed8792bacbaf669b28"} Jul 05 14:21:33.626425 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:33.626356 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" podStartSLOduration=1.6263436100000002 podStartE2EDuration="1.62634361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:21:33.626034974 +0000 UTC m=+2581.635504594" watchObservedRunningTime="2026-07-05 14:21:33.62634361 +0000 UTC m=+2581.635813231" Jul 05 14:21:43.308566 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:43.308487 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:43.308566 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:43.308537 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:43.320925 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:43.320899 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:21:43.664360 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:21:43.664280 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:22:12.507800 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:12.507768 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27"] Jul 05 14:22:27.528148 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.528090 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" podUID="2a6988cb-5b54-4d14-9339-b29a9d230aac" containerName="main" containerID="cri-o://e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1" gracePeriod=45 Jul 05 14:22:27.774592 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.774568 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:22:27.867849 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.867770 2569 generic.go:356] "Generic (PLEG): container finished" podID="2a6988cb-5b54-4d14-9339-b29a9d230aac" containerID="e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1" exitCode=0 Jul 05 14:22:27.867849 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.867818 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" Jul 05 14:22:27.868025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.867855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" event={"ID":"2a6988cb-5b54-4d14-9339-b29a9d230aac","Type":"ContainerDied","Data":"e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1"} Jul 05 14:22:27.868025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.867895 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27" event={"ID":"2a6988cb-5b54-4d14-9339-b29a9d230aac","Type":"ContainerDied","Data":"43cc336d141665ba02d9d883084f0207117b3acdf2580bed8792bacbaf669b28"} Jul 05 14:22:27.868025 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.867922 2569 scope.go:117] "RemoveContainer" containerID="e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1" Jul 05 14:22:27.876389 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.876342 2569 scope.go:117] "RemoveContainer" containerID="e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1" Jul 05 14:22:27.876640 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:22:27.876622 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1\": container with ID starting with e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1 not found: ID does not exist" containerID="e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1" Jul 05 14:22:27.876703 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.876648 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1"} err="failed to get container status \"e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1\": rpc error: code = NotFound desc = could not find container \"e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1\": container with ID starting with e30163678fdec6aa517b978a1dd89650cdec1ca3d3db50f6ea198418babd23c1 not found: ID does not exist" Jul 05 14:22:27.886901 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.886882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-dshm\") pod \"2a6988cb-5b54-4d14-9339-b29a9d230aac\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " Jul 05 14:22:27.886968 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.886961 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-home\") pod \"2a6988cb-5b54-4d14-9339-b29a9d230aac\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " Jul 05 14:22:27.887009 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.886989 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6988cb-5b54-4d14-9339-b29a9d230aac-tls-certs\") pod \"2a6988cb-5b54-4d14-9339-b29a9d230aac\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " Jul 05 14:22:27.887044 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887024 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xllqj\" (UniqueName: \"kubernetes.io/projected/2a6988cb-5b54-4d14-9339-b29a9d230aac-kube-api-access-xllqj\") pod \"2a6988cb-5b54-4d14-9339-b29a9d230aac\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " Jul 05 14:22:27.887085 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887043 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-tmp-dir\") pod \"2a6988cb-5b54-4d14-9339-b29a9d230aac\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " Jul 05 14:22:27.887124 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887092 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-model-cache\") pod \"2a6988cb-5b54-4d14-9339-b29a9d230aac\" (UID: \"2a6988cb-5b54-4d14-9339-b29a9d230aac\") " Jul 05 14:22:27.887296 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887247 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-home" (OuterVolumeSpecName: "home") pod "2a6988cb-5b54-4d14-9339-b29a9d230aac" (UID: "2a6988cb-5b54-4d14-9339-b29a9d230aac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:22:27.887413 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887327 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a6988cb-5b54-4d14-9339-b29a9d230aac" (UID: "2a6988cb-5b54-4d14-9339-b29a9d230aac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:22:27.887413 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887400 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a6988cb-5b54-4d14-9339-b29a9d230aac" (UID: "2a6988cb-5b54-4d14-9339-b29a9d230aac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:22:27.887532 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887518 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-home\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:22:27.887589 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887532 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-tmp-dir\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:22:27.887589 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.887541 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-model-cache\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:22:27.889054 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.889032 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-dshm" (OuterVolumeSpecName: "dshm") pod "2a6988cb-5b54-4d14-9339-b29a9d230aac" (UID: "2a6988cb-5b54-4d14-9339-b29a9d230aac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:22:27.889492 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.889473 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a6988cb-5b54-4d14-9339-b29a9d230aac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a6988cb-5b54-4d14-9339-b29a9d230aac" (UID: "2a6988cb-5b54-4d14-9339-b29a9d230aac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 05 14:22:27.889557 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.889484 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a6988cb-5b54-4d14-9339-b29a9d230aac-kube-api-access-xllqj" (OuterVolumeSpecName: "kube-api-access-xllqj") pod "2a6988cb-5b54-4d14-9339-b29a9d230aac" (UID: "2a6988cb-5b54-4d14-9339-b29a9d230aac"). InnerVolumeSpecName "kube-api-access-xllqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:22:27.988199 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.988171 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a6988cb-5b54-4d14-9339-b29a9d230aac-dshm\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:22:27.988199 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.988198 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a6988cb-5b54-4d14-9339-b29a9d230aac-tls-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:22:27.988341 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:27.988213 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xllqj\" (UniqueName: \"kubernetes.io/projected/2a6988cb-5b54-4d14-9339-b29a9d230aac-kube-api-access-xllqj\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:22:28.191394 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:28.191347 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27"] Jul 05 14:22:28.196059 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:28.196037 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-a7c515/rate-test-model-a7c515-kserve-7cdfbd5bcf-fvl27"] Jul 05 14:22:28.798724 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:28.798689 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a6988cb-5b54-4d14-9339-b29a9d230aac" path="/var/lib/kubelet/pods/2a6988cb-5b54-4d14-9339-b29a9d230aac/volumes" Jul 05 14:22:56.491502 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:56.491467 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7"] Jul 05 14:22:56.492057 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:22:56.491701 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" containerID="cri-o://0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4" gracePeriod=30 Jul 05 14:23:01.747700 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.747676 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:23:01.791519 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791441 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-credential-socket\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791645 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791531 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ccpqc\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-kube-api-access-ccpqc\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791645 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791567 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-token\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791645 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791640 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-socket\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791793 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791677 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-envoy\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791793 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791678 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:01.791793 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-podinfo\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791793 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791729 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-certs\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.791793 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791765 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istiod-ca-cert\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.792100 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.791805 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-data\") pod \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\" (UID: \"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d\") " Jul 05 14:23:01.792100 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.792048 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:01.792220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.792156 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.792220 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.792178 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.792329 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.792228 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:01.792329 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.792311 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:23:01.792887 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.792859 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:01.794277 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.794248 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-kube-api-access-ccpqc" (OuterVolumeSpecName: "kube-api-access-ccpqc") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "kube-api-access-ccpqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:23:01.794416 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.794278 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:23:01.794548 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.794521 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:23:01.794595 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.794574 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" (UID: "a3fdc9a7-189f-4baf-9383-395d4f8e1d6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:01.892760 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892734 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.892760 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892758 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.892889 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892768 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.892889 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892776 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.892889 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892784 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.892889 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892793 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccpqc\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-kube-api-access-ccpqc\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:01.892889 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:01.892802 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:02.032661 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.032623 2569 generic.go:356] "Generic (PLEG): container finished" podID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerID="0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4" exitCode=0 Jul 05 14:23:02.032910 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.032677 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" Jul 05 14:23:02.032910 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.032705 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" event={"ID":"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d","Type":"ContainerDied","Data":"0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4"} Jul 05 14:23:02.032910 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.032764 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7" event={"ID":"a3fdc9a7-189f-4baf-9383-395d4f8e1d6d","Type":"ContainerDied","Data":"505b73a313876d9980e530a6e5f42c6a8e2fb302bb91884159df9aeeeea32c2e"} Jul 05 14:23:02.032910 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.032794 2569 scope.go:117] "RemoveContainer" containerID="0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4" Jul 05 14:23:02.042088 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.042069 2569 scope.go:117] "RemoveContainer" containerID="eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e" Jul 05 14:23:02.050137 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.050120 2569 scope.go:117] "RemoveContainer" containerID="0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4" Jul 05 14:23:02.050420 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:23:02.050395 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4\": container with ID starting with 0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4 not found: ID does not exist" containerID="0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4" Jul 05 14:23:02.050510 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.050434 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4"} err="failed to get container status \"0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4\": rpc error: code = NotFound desc = could not find container \"0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4\": container with ID starting with 0636431e8e7e17ccdaf7dc5cf219f63995707d11634f41048d8a1571f932fce4 not found: ID does not exist" Jul 05 14:23:02.050510 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.050460 2569 scope.go:117] "RemoveContainer" containerID="eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e" Jul 05 14:23:02.050701 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:23:02.050685 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e\": container with ID starting with eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e not found: ID does not exist" containerID="eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e" Jul 05 14:23:02.050745 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.050706 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e"} err="failed to get container status \"eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e\": rpc error: code = NotFound desc = could not find container \"eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e\": container with ID starting with eb04a0aa3d2aade1cc717b109bc0f13776ae33b1ea96f42464864ec1b4c3608e not found: ID does not exist" Jul 05 14:23:02.057146 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.057123 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7"] Jul 05 14:23:02.058846 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.058825 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-9073be-openshift-default-5d9f6878cf-zzlq7"] Jul 05 14:23:02.800885 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:02.800852 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" path="/var/lib/kubelet/pods/a3fdc9a7-189f-4baf-9383-395d4f8e1d6d/volumes" Jul 05 14:23:03.319258 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:03.319228 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457"] Jul 05 14:23:03.319494 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:03.319453 2569 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" containerID="cri-o://6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133" gracePeriod=30 Jul 05 14:23:08.564071 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.564047 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:23:08.652492 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652410 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsrwb\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-kube-api-access-bsrwb\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652492 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652471 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-credential-socket\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652492 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652490 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-envoy\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652773 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652592 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-token\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652773 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652645 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-socket\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652773 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652677 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-certs\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652799 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-data\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652847 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istiod-ca-cert\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.652938 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652890 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-podinfo\") pod \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\" (UID: \"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43\") " Jul 05 14:23:08.653091 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652796 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:08.653091 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652956 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:08.653091 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.652945 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:08.653274 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.653254 2569 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-credential-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.653329 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.653287 2569 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-socket\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.653329 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.653304 2569 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-workload-certs\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.653329 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.653254 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 05 14:23:08.653670 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.653645 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-data" (OuterVolumeSpecName: "istio-data") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:08.654838 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.654810 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-token" (OuterVolumeSpecName: "istio-token") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:23:08.654959 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.654873 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 05 14:23:08.654959 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.654891 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-kube-api-access-bsrwb" (OuterVolumeSpecName: "kube-api-access-bsrwb") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "kube-api-access-bsrwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 05 14:23:08.654959 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.654906 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" (UID: "5ed16cfe-bbd2-4d0b-922a-c78f43fefb43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 05 14:23:08.753885 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.753853 2569 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-data\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.753885 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.753878 2569 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istiod-ca-cert\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.754049 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.753891 2569 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-podinfo\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.754049 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.753902 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsrwb\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-kube-api-access-bsrwb\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.754049 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.753915 2569 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-envoy\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:08.754049 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:08.753926 2569 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43-istio-token\") on node \"ip-10-0-130-182.ec2.internal\" DevicePath \"\"" Jul 05 14:23:09.067296 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.067261 2569 generic.go:356] "Generic (PLEG): container finished" podID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerID="6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133" exitCode=0 Jul 05 14:23:09.067485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.067307 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" Jul 05 14:23:09.067485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.067301 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" event={"ID":"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43","Type":"ContainerDied","Data":"6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133"} Jul 05 14:23:09.067485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.067424 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457" event={"ID":"5ed16cfe-bbd2-4d0b-922a-c78f43fefb43","Type":"ContainerDied","Data":"af4321cf652c6d3dc2df8f9c7a75676d736b5f1593a87f7441356a1e18bd8c2c"} Jul 05 14:23:09.067485 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.067448 2569 scope.go:117] "RemoveContainer" containerID="6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133" Jul 05 14:23:09.076190 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.076177 2569 scope.go:117] "RemoveContainer" containerID="b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc" Jul 05 14:23:09.083799 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.083778 2569 scope.go:117] "RemoveContainer" containerID="6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133" Jul 05 14:23:09.084142 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:23:09.084115 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133\": container with ID starting with 6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133 not found: ID does not exist" containerID="6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133" Jul 05 14:23:09.084248 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.084152 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133"} err="failed to get container status \"6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133\": rpc error: code = NotFound desc = could not find container \"6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133\": container with ID starting with 6d0cd99f68f28ab4790470b559e5874a36a91be0254c1215f67f97c01130e133 not found: ID does not exist" Jul 05 14:23:09.084248 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.084180 2569 scope.go:117] "RemoveContainer" containerID="b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc" Jul 05 14:23:09.084591 ip-10-0-130-182 kubenswrapper[2569]: E0705 14:23:09.084548 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc\": container with ID starting with b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc not found: ID does not exist" containerID="b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc" Jul 05 14:23:09.084591 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.084578 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc"} err="failed to get container status \"b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc\": rpc error: code = NotFound desc = could not find container \"b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc\": container with ID starting with b5ec16a5568bb6a9eec50ae14557c30fa46e2aa16de45d80a46eea08a9a3d9bc not found: ID does not exist" Jul 05 14:23:09.086708 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.086688 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457"] Jul 05 14:23:09.093390 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.092181 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-a7c515-openshift-default-845878bbbc-2g457"] Jul 05 14:23:09.559125 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.559094 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f79c67cfc-lpnrf_e152330b-409b-438e-acfd-b34426e9c030/copy-manifests/0.log" Jul 05 14:23:09.566894 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:09.566871 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f79c67cfc-lpnrf_e152330b-409b-438e-acfd-b34426e9c030/manager/0.log" Jul 05 14:23:10.799503 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:10.799467 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" path="/var/lib/kubelet/pods/5ed16cfe-bbd2-4d0b-922a-c78f43fefb43/volumes" Jul 05 14:23:11.419716 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:11.419689 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hcl5l_1d78f9b5-cf15-4df9-8ebe-28a1594a3b97/kuadrant-console-plugin/0.log" Jul 05 14:23:12.242967 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:12.242934 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678fpm6s_0b81a657-778a-458b-baec-0adbd300445c/istio-proxy/0.log" Jul 05 14:23:12.599539 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:12.599428 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68d7598454-ttp9l_c1064c7e-2d88-487b-b7ec-393a50a361ad/kube-auth-proxy/0.log" Jul 05 14:23:12.711865 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:12.711835 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/1.log" Jul 05 14:23:12.824599 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:12.824567 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z5jmr_733e1f4d-73ae-409a-8408-22589ea86b20/payload-pre-processing/0.log" Jul 05 14:23:13.057637 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:13.057610 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8446587799-l6kwv_6cdd610b-7966-4c3a-9648-21dbcf1828b2/router/0.log" Jul 05 14:23:13.732711 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:13.732683 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbv7k9t_d3820e1f-f601-4880-8133-beb9b244cd1d/main/0.log" Jul 05 14:23:13.961134 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:13.961109 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-qg2pw_96e584a0-fc7e-4e91-ab47-7f04a1ffe6b6/main/0.log" Jul 05 14:23:18.247610 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247577 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-489kn/must-gather-8gbf9"] Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247932 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247944 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247953 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247960 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247968 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247973 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247989 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a6988cb-5b54-4d14-9339-b29a9d230aac" containerName="main" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.247995 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a6988cb-5b54-4d14-9339-b29a9d230aac" containerName="main" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248046 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248054 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a6988cb-5b54-4d14-9339-b29a9d230aac" containerName="main" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248060 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" Jul 05 14:23:18.248086 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248067 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3fdc9a7-189f-4baf-9383-395d4f8e1d6d" containerName="istio-proxy" Jul 05 14:23:18.248687 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248124 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" Jul 05 14:23:18.248687 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248130 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" Jul 05 14:23:18.248687 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.248192 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ed16cfe-bbd2-4d0b-922a-c78f43fefb43" containerName="istio-proxy" Jul 05 14:23:18.251048 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.251028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.254125 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.254105 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-489kn\"/\"kube-root-ca.crt\"" Jul 05 14:23:18.255574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.255553 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-489kn\"/\"default-dockercfg-2cnkl\"" Jul 05 14:23:18.255574 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.255568 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-489kn\"/\"openshift-service-ca.crt\"" Jul 05 14:23:18.261344 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.261320 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-489kn/must-gather-8gbf9"] Jul 05 14:23:18.334838 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.334807 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c93de1dc-adda-47ea-ab76-962582ef619d-must-gather-output\") pod \"must-gather-8gbf9\" (UID: \"c93de1dc-adda-47ea-ab76-962582ef619d\") " pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.334997 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.334848 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zk4\" (UniqueName: \"kubernetes.io/projected/c93de1dc-adda-47ea-ab76-962582ef619d-kube-api-access-r5zk4\") pod \"must-gather-8gbf9\" (UID: \"c93de1dc-adda-47ea-ab76-962582ef619d\") " pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.435577 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.435534 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c93de1dc-adda-47ea-ab76-962582ef619d-must-gather-output\") pod \"must-gather-8gbf9\" (UID: \"c93de1dc-adda-47ea-ab76-962582ef619d\") " pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.435577 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.435584 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zk4\" (UniqueName: \"kubernetes.io/projected/c93de1dc-adda-47ea-ab76-962582ef619d-kube-api-access-r5zk4\") pod \"must-gather-8gbf9\" (UID: \"c93de1dc-adda-47ea-ab76-962582ef619d\") " pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.435868 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.435850 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/c93de1dc-adda-47ea-ab76-962582ef619d-must-gather-output\") pod \"must-gather-8gbf9\" (UID: \"c93de1dc-adda-47ea-ab76-962582ef619d\") " pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.444818 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.444786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zk4\" (UniqueName: \"kubernetes.io/projected/c93de1dc-adda-47ea-ab76-962582ef619d-kube-api-access-r5zk4\") pod \"must-gather-8gbf9\" (UID: \"c93de1dc-adda-47ea-ab76-962582ef619d\") " pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.560070 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.560008 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-489kn/must-gather-8gbf9" Jul 05 14:23:18.693457 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.693435 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-489kn/must-gather-8gbf9"] Jul 05 14:23:18.695323 ip-10-0-130-182 kubenswrapper[2569]: W0705 14:23:18.695300 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc93de1dc_adda_47ea_ab76_962582ef619d.slice/crio-c0201d7f9d4d11a83c47bc65c5b2bdf691765db78bbdada0b532c3b40e67d109 WatchSource:0}: Error finding container c0201d7f9d4d11a83c47bc65c5b2bdf691765db78bbdada0b532c3b40e67d109: Status 404 returned error can't find the container with id c0201d7f9d4d11a83c47bc65c5b2bdf691765db78bbdada0b532c3b40e67d109 Jul 05 14:23:18.697267 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:18.697252 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 05 14:23:19.116324 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:19.116288 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-489kn/must-gather-8gbf9" event={"ID":"c93de1dc-adda-47ea-ab76-962582ef619d","Type":"ContainerStarted","Data":"c0201d7f9d4d11a83c47bc65c5b2bdf691765db78bbdada0b532c3b40e67d109"} Jul 05 14:23:20.124901 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:20.124862 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-489kn/must-gather-8gbf9" event={"ID":"c93de1dc-adda-47ea-ab76-962582ef619d","Type":"ContainerStarted","Data":"c7142bab3a18596b4cdc76bc75b5b488b01cea4d74337204c066c2259b761bff"} Jul 05 14:23:20.125484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:20.125458 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-489kn/must-gather-8gbf9" event={"ID":"c93de1dc-adda-47ea-ab76-962582ef619d","Type":"ContainerStarted","Data":"119d9d9fc2433550905234c6cb3eea00de0d333a48b27eb69ac142385f58bae0"} Jul 05 14:23:20.141494 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:20.141442 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-489kn/must-gather-8gbf9" podStartSLOduration=1.336945413 podStartE2EDuration="2.141420057s" totalImagesPullingTime="804.474644ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-05 14:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:23:20.140944598 +0000 UTC m=+2688.150414220" watchObservedRunningTime="2026-07-05 14:23:20.141420057 +0000 UTC m=+2688.150889678" Jul 05 14:23:21.077116 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:21.077079 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-hgg8b_f2cb94bd-1a62-46db-b988-8726f95cea99/global-pull-secret-syncer/0.log" Jul 05 14:23:21.162192 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:21.162157 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vrzwb_6f72a14a-0f23-403a-8410-3085f801802e/konnectivity-agent/0.log" Jul 05 14:23:21.225005 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:21.224976 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-182.ec2.internal_bed7f726f53bfd3508e373fd03ee9ef0/haproxy/0.log" Jul 05 14:23:26.067847 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:26.067816 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-hcl5l_1d78f9b5-cf15-4df9-8ebe-28a1594a3b97/kuadrant-console-plugin/0.log" Jul 05 14:23:28.009529 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.009491 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-n6vq6_a53248a8-e057-4cc5-8535-10e1ecbd3a7e/cluster-monitoring-operator/0.log" Jul 05 14:23:28.039206 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.039174 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tsdgx_8c767b12-8a35-49e8-949c-6bb5264e0812/kube-state-metrics/0.log" Jul 05 14:23:28.061098 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.060932 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tsdgx_8c767b12-8a35-49e8-949c-6bb5264e0812/kube-rbac-proxy-main/0.log" Jul 05 14:23:28.085148 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.085123 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-tsdgx_8c767b12-8a35-49e8-949c-6bb5264e0812/kube-rbac-proxy-self/0.log" Jul 05 14:23:28.363482 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.363449 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m455r_e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46/node-exporter/0.log" Jul 05 14:23:28.395918 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.395854 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m455r_e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46/kube-rbac-proxy/0.log" Jul 05 14:23:28.422065 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.422032 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-m455r_e1f6fda1-1db9-4dfd-bc4a-5c8023ba3e46/init-textfile/0.log" Jul 05 14:23:28.484175 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.484093 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kvghg_100633fb-3a92-4178-9eb1-500bc480dbbc/kube-rbac-proxy-main/0.log" Jul 05 14:23:28.505129 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.505095 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kvghg_100633fb-3a92-4178-9eb1-500bc480dbbc/kube-rbac-proxy-self/0.log" Jul 05 14:23:28.529730 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.529694 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-kvghg_100633fb-3a92-4178-9eb1-500bc480dbbc/openshift-state-metrics/0.log" Jul 05 14:23:28.568941 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.568908 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/prometheus/0.log" Jul 05 14:23:28.591821 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.591795 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/config-reloader/0.log" Jul 05 14:23:28.614512 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.614447 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/thanos-sidecar/0.log" Jul 05 14:23:28.635607 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.635582 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/kube-rbac-proxy-web/0.log" Jul 05 14:23:28.659085 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.659036 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/kube-rbac-proxy/0.log" Jul 05 14:23:28.679848 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.679804 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/kube-rbac-proxy-thanos/0.log" Jul 05 14:23:28.701269 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:28.701172 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_76001d73-0111-4c67-b04f-0f04e63491cb/init-config-reloader/0.log" Jul 05 14:23:30.261402 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:30.261357 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-g8fr4_e45a90da-f828-44b2-9627-eb68a5f8e4c8/networking-console-plugin/0.log" Jul 05 14:23:31.033551 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.033510 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm"] Jul 05 14:23:31.040510 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.040479 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.046449 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.046425 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm"] Jul 05 14:23:31.176684 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.176648 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p22b\" (UniqueName: \"kubernetes.io/projected/1d463237-928b-47a4-a3a3-a7bd55716177-kube-api-access-7p22b\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.176959 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.176943 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-proc\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.177114 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.177100 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-podres\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.177294 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.177282 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-sys\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.177451 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.177437 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-lib-modules\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.278683 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.278643 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-sys\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279289 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279256 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-lib-modules\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279449 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279330 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p22b\" (UniqueName: \"kubernetes.io/projected/1d463237-928b-47a4-a3a3-a7bd55716177-kube-api-access-7p22b\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279449 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-sys\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279449 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279432 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-proc\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279626 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279442 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-lib-modules\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279626 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279479 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-podres\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279626 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279526 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-proc\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.279791 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.279649 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/1d463237-928b-47a4-a3a3-a7bd55716177-podres\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.288776 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.288704 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p22b\" (UniqueName: \"kubernetes.io/projected/1d463237-928b-47a4-a3a3-a7bd55716177-kube-api-access-7p22b\") pod \"perf-node-gather-daemonset-z2mnm\" (UID: \"1d463237-928b-47a4-a3a3-a7bd55716177\") " pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.356690 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.356651 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:31.406690 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.406661 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5b5cdf44d-9dmng_d555297d-8b75-40e5-af88-41e4f578bb4b/console/0.log" Jul 05 14:23:31.536717 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:31.536576 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm"] Jul 05 14:23:32.219229 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:32.219190 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" event={"ID":"1d463237-928b-47a4-a3a3-a7bd55716177","Type":"ContainerStarted","Data":"6337803ba1f1428daef96e2a3c016b768721ce1bf463ed20dfa07490d4457f07"} Jul 05 14:23:32.219471 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:32.219253 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" event={"ID":"1d463237-928b-47a4-a3a3-a7bd55716177","Type":"ContainerStarted","Data":"d4f58370c33c448de782211c8f85f3dbf0205a5628e881ddae943156c4b26107"} Jul 05 14:23:32.219630 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:32.219597 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:32.236802 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:32.236740 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" podStartSLOduration=1.236718897 podStartE2EDuration="1.236718897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-05 14:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-05 14:23:32.235728707 +0000 UTC m=+2700.245198327" watchObservedRunningTime="2026-07-05 14:23:32.236718897 +0000 UTC m=+2700.246188521" Jul 05 14:23:33.184192 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:33.184163 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lm677_8e146763-8e15-4664-b91d-b527657be7bb/dns/0.log" Jul 05 14:23:33.202467 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:33.202443 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lm677_8e146763-8e15-4664-b91d-b527657be7bb/kube-rbac-proxy/0.log" Jul 05 14:23:33.223201 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:33.223170 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8tp7h_82a3c6df-c47e-4738-a4fd-01179772eb73/dns-node-resolver/0.log" Jul 05 14:23:33.956793 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:33.956767 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zdz5r_cce826b0-2bf5-4efc-a2dc-ef404b5449cf/node-ca/0.log" Jul 05 14:23:34.872748 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:34.872716 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678fpm6s_0b81a657-778a-458b-baec-0adbd300445c/istio-proxy/0.log" Jul 05 14:23:35.101484 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:35.101459 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68d7598454-ttp9l_c1064c7e-2d88-487b-b7ec-393a50a361ad/kube-auth-proxy/0.log" Jul 05 14:23:35.130119 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:35.130040 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/0.log" Jul 05 14:23:35.166350 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:35.166316 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/1.log" Jul 05 14:23:35.197063 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:35.197031 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z5jmr_733e1f4d-73ae-409a-8408-22589ea86b20/payload-pre-processing/0.log" Jul 05 14:23:35.288183 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:35.288147 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8446587799-l6kwv_6cdd610b-7966-4c3a-9648-21dbcf1828b2/router/0.log" Jul 05 14:23:36.011361 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:36.011332 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jlm49_3730e0ef-cd9d-401d-ab49-9261e28e14b5/serve-healthcheck-canary/0.log" Jul 05 14:23:36.467773 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:36.467738 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xrkbr_054b55e9-2ef5-4320-ad4c-c25d2c221495/insights-operator/0.log" Jul 05 14:23:36.468142 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:36.468121 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-xrkbr_054b55e9-2ef5-4320-ad4c-c25d2c221495/insights-operator/1.log" Jul 05 14:23:36.618915 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:36.618876 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qv25g_78c320ec-5b42-4290-b828-cf373261e157/kube-rbac-proxy/0.log" Jul 05 14:23:36.639598 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:36.639569 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qv25g_78c320ec-5b42-4290-b828-cf373261e157/exporter/0.log" Jul 05 14:23:36.660052 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:36.660028 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qv25g_78c320ec-5b42-4290-b828-cf373261e157/extractor/0.log" Jul 05 14:23:39.240438 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:39.240412 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-489kn/perf-node-gather-daemonset-z2mnm" Jul 05 14:23:39.533244 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:39.533154 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f79c67cfc-lpnrf_e152330b-409b-438e-acfd-b34426e9c030/manager/0.log" Jul 05 14:23:39.555516 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:39.555487 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f79c67cfc-lpnrf_e152330b-409b-438e-acfd-b34426e9c030/copy-manifests/0.log" Jul 05 14:23:40.933080 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:40.933046 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-bf64756c4-7rbzh_4ece36f5-e414-4ba6-9167-8a0529f6f372/manager/0.log" Jul 05 14:23:40.977040 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:40.976967 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678fpm6s_0b81a657-778a-458b-baec-0adbd300445c/istio-proxy/0.log" Jul 05 14:23:41.190308 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:41.190273 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68d7598454-ttp9l_c1064c7e-2d88-487b-b7ec-393a50a361ad/kube-auth-proxy/0.log" Jul 05 14:23:41.220551 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:41.220520 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/0.log" Jul 05 14:23:41.260774 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:41.260700 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/1.log" Jul 05 14:23:41.287985 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:41.287955 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z5jmr_733e1f4d-73ae-409a-8408-22589ea86b20/payload-pre-processing/0.log" Jul 05 14:23:41.370904 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:41.370873 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8446587799-l6kwv_6cdd610b-7966-4c3a-9648-21dbcf1828b2/router/0.log" Jul 05 14:23:42.668509 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:42.668482 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678fpm6s_0b81a657-778a-458b-baec-0adbd300445c/istio-proxy/0.log" Jul 05 14:23:42.887579 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:42.887554 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68d7598454-ttp9l_c1064c7e-2d88-487b-b7ec-393a50a361ad/kube-auth-proxy/0.log" Jul 05 14:23:42.909073 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:42.909033 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/0.log" Jul 05 14:23:42.951016 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:42.950987 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-pnxpz_aa1dd6b7-4688-4c89-87bf-2c49e3535502/istio-proxy/1.log" Jul 05 14:23:42.977521 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:42.977489 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-z5jmr_733e1f4d-73ae-409a-8408-22589ea86b20/payload-pre-processing/0.log" Jul 05 14:23:43.065011 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:43.064987 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8446587799-l6kwv_6cdd610b-7966-4c3a-9648-21dbcf1828b2/router/0.log" Jul 05 14:23:46.008464 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:46.008432 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-wmd89_70591d88-9268-44bc-bcba-453202d409ba/migrator/0.log" Jul 05 14:23:46.030651 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:46.030617 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-wmd89_70591d88-9268-44bc-bcba-453202d409ba/graceful-termination/0.log" Jul 05 14:23:46.377490 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:46.377411 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-4d8vb_19560aa3-2a88-468c-aa1c-0320e499a8ff/kube-storage-version-migrator-operator/1.log" Jul 05 14:23:46.378330 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:46.378312 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-4d8vb_19560aa3-2a88-468c-aa1c-0320e499a8ff/kube-storage-version-migrator-operator/0.log" Jul 05 14:23:47.315138 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.315109 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/kube-multus-additional-cni-plugins/0.log" Jul 05 14:23:47.335894 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.335871 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/egress-router-binary-copy/0.log" Jul 05 14:23:47.355989 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.355970 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/cni-plugins/0.log" Jul 05 14:23:47.376157 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.376137 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/bond-cni-plugin/0.log" Jul 05 14:23:47.394843 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.394826 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/routeoverride-cni/0.log" Jul 05 14:23:47.414170 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.414154 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/whereabouts-cni-bincopy/0.log" Jul 05 14:23:47.437210 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.437179 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7229l_b8f07c2e-c429-43df-840a-b38476bd199c/whereabouts-cni/0.log" Jul 05 14:23:47.773568 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.773538 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jq5sl_e7f25710-f553-4569-8644-826c6adb6727/kube-multus/0.log" Jul 05 14:23:47.830259 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.830198 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jbfxn_1d342df4-57dc-415e-9232-a44b96de4d37/network-metrics-daemon/0.log" Jul 05 14:23:47.850029 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:47.850005 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-jbfxn_1d342df4-57dc-415e-9232-a44b96de4d37/kube-rbac-proxy/0.log" Jul 05 14:23:48.756096 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.756068 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/ovn-controller/0.log" Jul 05 14:23:48.772924 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.772903 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/ovn-acl-logging/0.log" Jul 05 14:23:48.802325 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.802301 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/kube-rbac-proxy-node/0.log" Jul 05 14:23:48.824129 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.824107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/kube-rbac-proxy-ovn-metrics/0.log" Jul 05 14:23:48.841695 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.841674 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/northd/0.log" Jul 05 14:23:48.875851 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.875832 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/nbdb/0.log" Jul 05 14:23:48.896652 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.896627 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/sbdb/0.log" Jul 05 14:23:48.956161 ip-10-0-130-182 kubenswrapper[2569]: I0705 14:23:48.956134 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-cgzlg_9694b03a-00a4-4b05-a482-fb25462145f1/ovnkube-controller/0.log"