Jul 06 16:21:51.017234 ip-10-0-136-202 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 06 16:21:51.017248 ip-10-0-136-202 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 06 16:21:51.017257 ip-10-0-136-202 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 06 16:21:51.017560 ip-10-0-136-202 systemd[1]: Failed to start Kubernetes Kubelet. Jul 06 16:22:01.210626 ip-10-0-136-202 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 06 16:22:01.210642 ip-10-0-136-202 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 0f7d85b0dc154aa58637365db5b7592c -- Jul 06 16:23:36.144551 ip-10-0-136-202 systemd[1]: Starting Kubernetes Kubelet... Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.583197 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588181 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588311 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588318 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588324 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588327 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588330 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588333 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:23:36.608143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588336 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588338 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588341 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588343 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588347 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588351 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588355 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588357 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588360 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588365 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588369 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588372 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588374 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588377 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588380 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588383 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588386 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588389 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588392 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588395 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:23:36.610210 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588398 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588403 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588406 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588408 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588411 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588415 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588419 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588422 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588425 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588428 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588431 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588434 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588438 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588443 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588446 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588449 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588452 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588455 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588457 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:23:36.611742 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588460 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588463 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588465 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588468 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588470 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588473 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588478 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588480 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588483 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588486 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588488 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588491 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588495 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588498 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588501 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588504 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588507 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588509 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588512 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588514 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:23:36.613658 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588518 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588520 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588523 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588526 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588528 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588531 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588533 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588536 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588538 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588541 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588544 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588546 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588549 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588552 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588555 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588559 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588562 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588566 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588569 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588574 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:23:36.628378 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588578 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588581 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588584 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588587 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588590 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588593 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588596 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588599 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588602 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588605 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588608 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588611 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.588614 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589357 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589364 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589367 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589371 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589374 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589377 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589381 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:23:36.644385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589384 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589386 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589389 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589392 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589395 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589398 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589400 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589403 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589405 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589408 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589411 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589413 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589416 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589419 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589422 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589425 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589430 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589433 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589436 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589439 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:23:36.691162 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589442 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589444 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589447 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589450 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589452 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589455 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589458 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589461 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589464 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589466 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589470 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589473 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589476 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589478 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589481 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589484 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589486 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589489 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589491 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589494 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:23:36.709468 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589496 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:23:36.664007 ip-10-0-136-202 systemd[1]: Started Kubernetes Kubelet. Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589499 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589501 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589506 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589509 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589512 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589515 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589518 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589524 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589526 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589529 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589532 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589535 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589539 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589543 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589546 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589548 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589551 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589553 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:23:36.723575 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589556 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589559 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589561 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589566 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589568 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589572 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589574 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589577 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589579 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589582 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589585 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589587 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589591 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589593 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589596 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589598 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589601 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589603 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589606 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589609 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:23:36.771521 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589611 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589615 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589617 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589620 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589622 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589626 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589629 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589632 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589634 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589636 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589640 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589644 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.589646 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590023 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590028 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590033 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590036 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590039 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590041 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:23:36.906623 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590044 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590046 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590049 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590051 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590054 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590056 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590059 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590062 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590065 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590067 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590070 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590073 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590075 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590078 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590082 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590085 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590087 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590090 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590092 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590095 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:23:36.907673 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590098 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590100 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590103 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590105 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590108 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590110 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590113 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590115 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590119 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590122 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590124 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590127 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590129 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590132 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590134 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590137 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590141 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590143 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590146 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:23:36.908513 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590148 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590151 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590153 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590156 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590159 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590162 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590164 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590167 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590170 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590174 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590177 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590180 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590183 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590186 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590188 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590191 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590193 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590195 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590198 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590200 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:23:36.909143 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590212 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590216 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590237 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590240 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590243 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590245 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590248 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590251 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590253 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590256 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590258 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590261 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590263 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590266 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590270 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590273 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590275 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590278 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590281 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:23:36.910551 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590285 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590289 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590293 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590295 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590298 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590301 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590303 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590306 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590308 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590311 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590313 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590316 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590318 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590321 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.590325 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590395 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590402 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590410 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590417 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590423 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590428 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 06 16:23:36.911178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590434 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590439 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590442 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590445 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590449 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590452 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590455 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590458 2563 flags.go:64] FLAG: --cgroup-root="" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590461 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590464 2563 flags.go:64] FLAG: --client-ca-file="" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590468 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590473 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590477 2563 flags.go:64] FLAG: --cluster-domain="" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590480 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590484 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590487 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590490 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590494 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590497 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590500 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590503 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590506 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590509 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590512 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 06 16:23:36.911863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590515 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590519 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590523 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590526 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590529 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590532 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590535 2563 flags.go:64] FLAG: --enable-server="true" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590538 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590542 2563 flags.go:64] FLAG: --event-burst="100" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590545 2563 flags.go:64] FLAG: --event-qps="50" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590548 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590551 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590554 2563 flags.go:64] FLAG: --eviction-hard="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590558 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590560 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590563 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590566 2563 flags.go:64] FLAG: --eviction-soft="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590569 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590572 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590578 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590581 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590584 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590587 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590590 2563 flags.go:64] FLAG: --feature-gates="" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590594 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 06 16:23:36.913680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590596 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590599 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590603 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590606 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590609 2563 flags.go:64] FLAG: --help="false" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590611 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-136-202.ec2.internal" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590614 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590617 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590621 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590625 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590628 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590631 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590634 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590636 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590639 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590642 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590645 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590649 2563 flags.go:64] FLAG: --kube-reserved="" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590651 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590654 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590657 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590660 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590662 2563 flags.go:64] FLAG: --lock-file="" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590665 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 06 16:23:36.914534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590668 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590672 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590678 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590682 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590684 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590687 2563 flags.go:64] FLAG: --logging-format="text" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590690 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590693 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590696 2563 flags.go:64] FLAG: --manifest-url="" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590699 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590703 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590706 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590710 2563 flags.go:64] FLAG: --max-pods="110" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590713 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590716 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590719 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590723 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590725 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590728 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590731 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590739 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590742 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590745 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 06 16:23:37.070256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590747 2563 flags.go:64] FLAG: --pod-cidr="" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590750 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590756 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590759 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590762 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590765 2563 flags.go:64] FLAG: --port="10250" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590768 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590771 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00b66cc9ef96cd8a2" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590774 2563 flags.go:64] FLAG: --qos-reserved="" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590777 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590780 2563 flags.go:64] FLAG: --register-node="true" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590783 2563 flags.go:64] FLAG: --register-with-taints="" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590794 2563 flags.go:64] FLAG: --registry-burst="10" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590798 2563 flags.go:64] FLAG: --registry-qps="5" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590801 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590804 2563 flags.go:64] FLAG: --reserved-memory="" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590807 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590810 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590813 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590816 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590818 2563 flags.go:64] FLAG: --runonce="false" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590821 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590824 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590827 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590830 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 06 16:23:37.071092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590834 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590837 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590840 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590843 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590846 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590849 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590852 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590854 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590857 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590860 2563 flags.go:64] FLAG: --system-cgroups="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590863 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590868 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590871 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590874 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590878 2563 flags.go:64] FLAG: --tls-min-version="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590881 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590884 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590886 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590889 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590893 2563 flags.go:64] FLAG: --v="2" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590898 2563 flags.go:64] FLAG: --version="false" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590902 2563 flags.go:64] FLAG: --vmodule="" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590906 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.590909 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591053 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:23:37.071835 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591057 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591060 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591063 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591066 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591069 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591072 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591074 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591079 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591081 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591084 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591086 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591089 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591092 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591094 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591097 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591099 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591102 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591105 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591108 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591111 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:23:37.072597 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591113 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591116 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591119 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591121 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591124 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591127 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591131 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591134 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591137 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591140 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591142 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591145 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591148 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591152 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591155 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591159 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591161 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591164 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:23:37.073132 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591168 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591174 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591177 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591180 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591183 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591185 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591188 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591191 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591194 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591197 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591199 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591202 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591205 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591207 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591210 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591213 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591215 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591218 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591235 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:23:37.073683 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591238 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591242 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591245 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591247 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591251 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591255 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591258 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591260 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591263 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591266 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591268 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591271 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591274 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591276 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591281 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591283 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591286 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591288 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591291 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591293 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:23:37.074184 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591296 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591298 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591301 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591303 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591306 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591308 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591311 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591313 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591317 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591321 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591323 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591326 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591329 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591332 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591335 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591338 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591340 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591343 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591349 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591352 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:23:37.074772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.591354 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.591955 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.598686 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.598711 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.599758 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.599765 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599821 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599827 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599830 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599833 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599836 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599838 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599841 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599844 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599847 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:23:37.075306 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599850 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599852 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599855 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599857 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599860 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599863 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599865 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599868 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599870 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599873 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599876 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599880 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599884 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599887 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599890 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599897 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599900 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599902 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599905 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599909 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:23:37.075808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599911 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599914 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599917 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599920 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599923 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599926 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599928 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599931 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599934 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599937 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599940 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599942 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599945 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599950 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599952 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599955 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599958 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599960 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599963 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:23:37.076359 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599966 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599970 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599974 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599977 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599980 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599982 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599985 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599988 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599990 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599994 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.599996 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600013 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600017 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600020 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600023 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600026 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600029 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600032 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600035 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:23:37.076892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600037 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600040 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600043 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600045 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600048 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600051 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600054 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600059 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600063 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600066 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600069 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600072 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600074 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600077 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600079 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600082 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600084 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600087 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600090 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:23:37.077412 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600092 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600095 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600097 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600100 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600104 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600107 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600109 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600112 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600115 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600117 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600120 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600123 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600125 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.600130 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600261 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 06 16:23:37.077941 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600267 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600270 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600272 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600275 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600278 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600282 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600285 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600287 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600290 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600294 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600298 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600301 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600304 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600306 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600309 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600312 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600314 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600317 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600320 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 06 16:23:37.078376 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600322 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600325 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600328 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600331 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600333 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600336 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600338 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600341 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600343 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600346 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600348 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600351 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600353 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600356 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600358 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600360 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600365 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600368 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600374 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600377 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 06 16:23:37.078909 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600379 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600382 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600384 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600387 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600390 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600392 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600395 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600397 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600400 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600402 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600405 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600408 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600411 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600415 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600419 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600422 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600425 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600427 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600430 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600433 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 06 16:23:37.079445 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600435 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600438 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600440 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600443 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600446 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600448 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600451 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600453 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600457 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600460 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600463 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600466 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600468 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600471 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600473 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600476 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600478 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600481 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600483 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600486 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 06 16:23:37.079996 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600488 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600491 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600493 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600496 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600498 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600501 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600503 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600506 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600508 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600511 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600514 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600516 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600520 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600523 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600526 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600529 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600532 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600534 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 06 16:23:37.080541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:36.600537 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.600542 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.600639 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.603079 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.604462 2563 server.go:1019] "Starting client certificate rotation" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.604557 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.605269 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.607168 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.611878 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.624624 2563 log.go:25] "Validated CRI v1 runtime API" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.627162 2563 log.go:25] "Validated CRI v1 image API" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.628837 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 06 16:23:37.081046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.632235 2563 fs.go:135] Filesystem UUIDs: map[08efc4c1-144e-4e65-a9a4-7f00e0c259b6:/dev/nvme0n1p3 5db5f5c4-a803-4402-8d81-282d1b31a7ed:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2] Jul 06 16:23:37.081700 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.632256 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.638735 2563 manager.go:216] Machine: {Timestamp:2026-07-06 16:23:36.637333568 +0000 UTC m=+0.358121609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100163 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21c173924aada3b48d7c795cf459b5 SystemUUID:ec21c173-924a-ada3-b48d-7c795cf459b5 BootID:0f7d85b0-dc15-4aa5-8637-365db5b7592c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:95:07:62:96:23 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:95:07:62:96:23 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:62:8f:f9:fb:91 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.638840 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.638925 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.642711 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.642746 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.643530 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.643541 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 06 16:23:37.232875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.643549 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.645331 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.646515 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649561 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649678 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649706 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649716 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649729 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649739 2563 kubelet.go:398] "Adding apiserver pod source" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.649754 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.650125 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.650140 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.651713 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.652545 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.654528 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.654556 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656282 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656294 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656300 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656307 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656316 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656322 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656331 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656336 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656343 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656348 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656356 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.656361 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.658066 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.658085 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.662980 2563 server.go:1268] "Started kubelet" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.663057 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.663156 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.663099 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.664706 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.664890 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.664914 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.664936 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.665854 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 06 16:23:37.239375 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.668699 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.669863 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.669871 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.669965 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.670447 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.670451 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.670464 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.670714 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.670725 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671103 2563 factory.go:153] Registering CRI-O factory Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671115 2563 factory.go:223] Registration of the crio container factory successfully Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671195 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671206 2563 factory.go:55] Registering systemd factory Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671212 2563 factory.go:223] Registration of the systemd container factory successfully Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671269 2563 factory.go:103] Registering Raw factory Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671280 2563 manager.go:1196] Started watching for new ooms in manager Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.671775 2563 manager.go:349] Starting recovery of all containers Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.677969 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.678166 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.678256 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff0649b237f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.662950783 +0000 UTC m=+0.383738802,LastTimestamp:2026-07-06 16:23:36.662950783 +0000 UTC m=+0.383738802,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.689819 2563 manager.go:354] Recovery completed Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.696360 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698186 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698247 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698260 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698770 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698779 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698794 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.698829 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.699513 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.699538 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.699549 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.699931 2563 policy_none.go:49] "None policy: Start" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.699944 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 06 16:23:37.240436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.699953 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 06 16:23:37.241339 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.700872 2563 policy_none.go:47] "Start" Jul 06 16:23:37.241339 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.701539 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff066b5435d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-202.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.698217309 +0000 UTC m=+0.419005328,LastTimestamp:2026-07-06 16:23:36.698217309 +0000 UTC m=+0.419005328,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.241339 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.712728 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff066b5d3fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-202.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.698254333 +0000 UTC m=+0.419042351,LastTimestamp:2026-07-06 16:23:36.698254333 +0000 UTC m=+0.419042351,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.241339 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.724661 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff066b5f79f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-202.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.698263455 +0000 UTC m=+0.419051474,LastTimestamp:2026-07-06 16:23:36.698263455 +0000 UTC m=+0.419051474,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.241339 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.737072 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-202.ec2.internal.18bfbff066b5435d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff066b5435d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-202.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.698217309 +0000 UTC m=+0.419005328,LastTimestamp:2026-07-06 16:23:36.699522821 +0000 UTC m=+0.420310849,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.751384 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-202.ec2.internal.18bfbff066b5d3fd\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff066b5d3fd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-202.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.698254333 +0000 UTC m=+0.419042351,LastTimestamp:2026-07-06 16:23:36.699543675 +0000 UTC m=+0.420331693,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.756131 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gvf7m" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.762043 2563 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-136-202.ec2.internal.18bfbff066b5f79f\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-202.ec2.internal.18bfbff066b5f79f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-202.ec2.internal,UID:ip-10-0-136-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-202.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-202.ec2.internal,},FirstTimestamp:2026-07-06 16:23:36.698263455 +0000 UTC m=+0.419051474,LastTimestamp:2026-07-06 16:23:36.699552663 +0000 UTC m=+0.420340683,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-202.ec2.internal,}" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.770183 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.770285 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gvf7m" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.779950 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.781095 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.781127 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.781147 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.781175 2563 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.783898 2563 manager.go:337] "Starting Device Plugin manager" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.783941 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.783958 2563 server.go:87] "Starting device plugin registration server" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.784248 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.784262 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.784523 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.784584 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.784590 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.785534 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.785705 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.795806 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.881706 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal"] Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.881782 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.241573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.883364 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.883397 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.883416 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.884489 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.884596 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.884665 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.884683 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885191 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885231 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885193 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885193 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885256 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885259 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885268 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885274 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885243 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.885407 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.886544 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.886567 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.887189 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientMemory" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.887203 2563 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-136-202.ec2.internal\" not found" node="ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.887209 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasNoDiskPressure" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.887257 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeHasSufficientPID" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.897153 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.897169 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-202.ec2.internal\": node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.919797 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-202.ec2.internal\" not found" node="ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.924037 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-202.ec2.internal\" not found" node="ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:36.924364 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.972061 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35c3367aea0364e635c7dba529cb3ff-config\") pod \"kube-apiserver-proxy-ip-10-0-136-202.ec2.internal\" (UID: \"a35c3367aea0364e635c7dba529cb3ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.242266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.972084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/897d6bcf4199302dfb88bc40b83b5bcb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal\" (UID: \"897d6bcf4199302dfb88bc40b83b5bcb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:36.972101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/897d6bcf4199302dfb88bc40b83b5bcb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal\" (UID: \"897d6bcf4199302dfb88bc40b83b5bcb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.025248 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.072217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35c3367aea0364e635c7dba529cb3ff-config\") pod \"kube-apiserver-proxy-ip-10-0-136-202.ec2.internal\" (UID: \"a35c3367aea0364e635c7dba529cb3ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.072265 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/897d6bcf4199302dfb88bc40b83b5bcb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal\" (UID: \"897d6bcf4199302dfb88bc40b83b5bcb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.072290 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/897d6bcf4199302dfb88bc40b83b5bcb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal\" (UID: \"897d6bcf4199302dfb88bc40b83b5bcb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.072326 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a35c3367aea0364e635c7dba529cb3ff-config\") pod \"kube-apiserver-proxy-ip-10-0-136-202.ec2.internal\" (UID: \"a35c3367aea0364e635c7dba529cb3ff\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.072333 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/897d6bcf4199302dfb88bc40b83b5bcb-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal\" (UID: \"897d6bcf4199302dfb88bc40b83b5bcb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.072338 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/897d6bcf4199302dfb88bc40b83b5bcb-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal\" (UID: \"897d6bcf4199302dfb88bc40b83b5bcb\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.126086 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.223815 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.226965 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.243047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.226983 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" Jul 06 16:23:37.404573 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.328022 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.597978 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.428551 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:37.597978 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.528980 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.603441 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:37.603582 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:37.629732 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:38.735613 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-202.ec2.internal\" not found" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.738785 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.776140 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.787426 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-05 16:18:37 +0000 UTC" deadline="2028-04-01 20:58:54.339731178 +0000 UTC" Jul 06 16:23:38.787642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.787456 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15244h35m15.552278873s" Jul 06 16:23:38.935868 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.796825 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 16:23:38.935868 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.798706 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" Jul 06 16:23:38.935868 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.809730 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 06 16:23:38.935868 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.810745 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:23:38.935868 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.934859 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:23:39.094238 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:38.966566 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:23:39.457543 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:39.296186 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897d6bcf4199302dfb88bc40b83b5bcb.slice/crio-fc0798371ab92713360567a65e510d709a6433a8747d33eb693d90211f1f73cc WatchSource:0}: Error finding container fc0798371ab92713360567a65e510d709a6433a8747d33eb693d90211f1f73cc: Status 404 returned error can't find the container with id fc0798371ab92713360567a65e510d709a6433a8747d33eb693d90211f1f73cc Jul 06 16:23:39.668812 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.590183 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 06 16:23:39.668812 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.656594 2563 apiserver.go:52] "Watching apiserver" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.674509 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.674850 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-hhjqj","kube-system/global-pull-secret-syncer-m285v","kube-system/konnectivity-agent-zhcw4","kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal","openshift-multus/multus-additional-cni-plugins-57tjt","openshift-network-diagnostics/network-check-target-xxtrg","openshift-ovn-kubernetes/ovnkube-node-56nhz","openshift-cluster-node-tuning-operator/tuned-hmk6v","openshift-dns/node-resolver-nm6dl","openshift-image-registry/node-ca-796kr","openshift-multus/multus-p4mwp","openshift-multus/network-metrics-daemon-d6nms"] Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.676200 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.676368 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.677703 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.678810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.680186 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.681307 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.682672 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.684011 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.684085 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.685137 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.686436 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.687728 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.689261 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.689312 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.690041 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.690058 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.690068 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.690616 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hv92n\"" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.691320 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.691521 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.691559 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:39.797860 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.692024 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.693126 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-systemd\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694808 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-etc-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694835 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-cni-bin\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694880 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694904 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-systemd-units\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694925 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-ovn\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694946 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-log-socket\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-ovnkube-script-lib\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.694991 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-slash\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-run-ovn-kubernetes\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695032 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-ovnkube-config\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.799681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695059 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-env-overrides\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695081 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5afa0111-b488-47f0-b10e-2c657d2f9057-ovn-node-metrics-cert\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695113 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rr89\" (UniqueName: \"kubernetes.io/projected/5afa0111-b488-47f0-b10e-2c657d2f9057-kube-api-access-4rr89\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695140 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-kubelet\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695162 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-var-lib-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-node-log\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695210 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-cni-netd\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.695252 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-run-netns\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.700542 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.701908 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.702099 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ljtkh\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.702126 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.702213 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.702220 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pkzkx\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.702273 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.702211 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.703832 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.703887 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.703914 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.703974 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.704136 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ldrvh\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.704164 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705278 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.800533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705384 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-k5kdj\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705469 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705580 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7f9px\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705640 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705673 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-vstl9\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705673 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hc6tv\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705767 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bnnq4\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705823 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705896 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.705947 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.706146 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.706160 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.706257 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.706372 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.731934 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.739708 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2s9nx" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.756803 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2s9nx" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.777353 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.792426 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" event={"ID":"897d6bcf4199302dfb88bc40b83b5bcb","Type":"ContainerStarted","Data":"fc0798371ab92713360567a65e510d709a6433a8747d33eb693d90211f1f73cc"} Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795654 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65242094-074d-41d5-a8e9-d22a5c636b9f-dbus\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795681 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-run-netns\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.801647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795747 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-hostroot\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795768 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795811 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-var-lib-kubelet\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795822 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-run-netns\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795839 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.795860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796319 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-ovn\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796341 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-log-socket\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-ovnkube-script-lib\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796377 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8qnm\" (UniqueName: \"kubernetes.io/projected/fd336a71-4bf2-4084-af55-e8a606870ddb-kube-api-access-f8qnm\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796404 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-systemd\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-run\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796427 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-ovn\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.802749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796453 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-log-socket\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.804475 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796482 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rr89\" (UniqueName: \"kubernetes.io/projected/5afa0111-b488-47f0-b10e-2c657d2f9057-kube-api-access-4rr89\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.866211 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.822907 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" event={"ID":"a35c3367aea0364e635c7dba529cb3ff","Type":"ContainerStarted","Data":"feffda994b5f17c559aaf2b865ce1944b2f43c3bb4120a8efa0665cdb2e7265c"} Jul 06 16:23:39.866211 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.796506 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-socket-dir-parent\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.866211 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.823010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd336a71-4bf2-4084-af55-e8a606870ddb-host-slash\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:39.866211 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.823038 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysconfig\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.866211 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.823139 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-ovnkube-script-lib\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888047 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888399 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-cni-binary-copy\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888465 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-multus-certs\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888499 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9zfj\" (UniqueName: \"kubernetes.io/projected/2115fb78-bf70-4dfc-8365-5caab1a40403-kube-api-access-f9zfj\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888529 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-tmp-dir\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cndmb\" (UniqueName: \"kubernetes.io/projected/bede19a7-4918-4b48-8bf0-f5ade4d53795-kube-api-access-cndmb\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888581 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-host\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888609 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6rh\" (UniqueName: \"kubernetes.io/projected/03734ae6-5b46-440f-99a2-6eac216a3675-kube-api-access-nv6rh\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888945 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-systemd\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.888977 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-hosts-file\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889005 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65242094-074d-41d5-a8e9-d22a5c636b9f-kubelet-config\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889031 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-modprobe-d\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889062 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.965754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889100 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ws2ss\" (UniqueName: \"kubernetes.io/projected/0f864ebe-e103-4a19-96c9-cd9d2beba99f-kube-api-access-ws2ss\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889125 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-run-systemd\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889130 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-cnibin\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889193 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2115fb78-bf70-4dfc-8365-5caab1a40403-cni-binary-copy\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-ovnkube-config\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889392 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-env-overrides\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-system-cni-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.889456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-node-log\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.890543 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-env-overrides\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.890701 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5afa0111-b488-47f0-b10e-2c657d2f9057-ovnkube-config\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.890852 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-node-log\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.892927 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-cni-multus\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.892972 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-daemon-config\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893033 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bede19a7-4918-4b48-8bf0-f5ade4d53795-serviceca\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:39.967045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893064 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-lib-modules\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893090 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5x8s\" (UniqueName: \"kubernetes.io/projected/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-kube-api-access-g5x8s\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-system-cni-dir\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893142 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-device-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-etc-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-cni-bin\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893289 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-etc-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893332 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-cni-bin\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893379 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-os-release\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-cni-bin\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893466 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-sys\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893493 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-tuned\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5f6j\" (UniqueName: \"kubernetes.io/projected/1aa4a323-253b-4c0e-907e-e8521c1c418f-kube-api-access-g5f6j\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.967923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893545 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-k8s-cni-cncf-io\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893577 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-conf-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893605 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdhhl\" (UniqueName: \"kubernetes.io/projected/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-kube-api-access-fdhhl\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893650 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-socket-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893837 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-sys-fs\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.893875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8a26bd-3e9b-4308-915d-f6fc9cd8394a-agent-certs\") pod \"konnectivity-agent-zhcw4\" (UID: \"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a\") " pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894333 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-kubernetes\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-os-release\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-kubelet\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894434 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-cni-netd\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894457 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-kubelet\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894480 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysctl-d\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894507 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894521 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-cni-netd\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.969018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894543 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-registration-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894567 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894573 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-kubelet\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894601 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-systemd-units\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894613 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-etc-kubernetes\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894637 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bede19a7-4918-4b48-8bf0-f5ade4d53795-host\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894678 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-slash\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894702 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-run-ovn-kubernetes\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894736 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5afa0111-b488-47f0-b10e-2c657d2f9057-ovn-node-metrics-cert\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894744 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-slash\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-netns\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894815 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-systemd-units\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894815 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8a26bd-3e9b-4308-915d-f6fc9cd8394a-konnectivity-ca\") pod \"konnectivity-agent-zhcw4\" (UID: \"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a\") " pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:39.970255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894854 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd336a71-4bf2-4084-af55-e8a606870ddb-iptables-alerter-script\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894880 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-host-run-ovn-kubernetes\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894918 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-tmp\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.894976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-var-lib-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.895024 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5afa0111-b488-47f0-b10e-2c657d2f9057-var-lib-openvswitch\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.895058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-cni-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.895085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysctl-conf\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.895105 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-cnibin\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.896786 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rr89\" (UniqueName: \"kubernetes.io/projected/5afa0111-b488-47f0-b10e-2c657d2f9057-kube-api-access-4rr89\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:39.971367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.897774 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5afa0111-b488-47f0-b10e-2c657d2f9057-ovn-node-metrics-cert\") pod \"ovnkube-node-56nhz\" (UID: \"5afa0111-b488-47f0-b10e-2c657d2f9057\") " pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.995981 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-modprobe-d\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996018 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ws2ss\" (UniqueName: \"kubernetes.io/projected/0f864ebe-e103-4a19-96c9-cd9d2beba99f-kube-api-access-ws2ss\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-cnibin\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996092 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2115fb78-bf70-4dfc-8365-5caab1a40403-cni-binary-copy\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996115 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-system-cni-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-cni-multus\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996158 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-daemon-config\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996179 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bede19a7-4918-4b48-8bf0-f5ade4d53795-serviceca\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996203 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-lib-modules\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996225 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-tuning-conf-dir\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996240 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-system-cni-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5x8s\" (UniqueName: \"kubernetes.io/projected/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-kube-api-access-g5x8s\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996201 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-modprobe-d\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.050435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996295 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-system-cni-dir\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996298 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-cni-multus\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996322 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-device-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996348 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-os-release\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996363 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-cnibin\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-cni-bin\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996407 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-system-cni-dir\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996417 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-sys\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996451 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-device-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996464 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-tuned\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996484 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5f6j\" (UniqueName: \"kubernetes.io/projected/1aa4a323-253b-4c0e-907e-e8521c1c418f-kube-api-access-g5f6j\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996500 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-k8s-cni-cncf-io\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996503 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-os-release\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996370 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-lib-modules\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.052658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996518 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-conf-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdhhl\" (UniqueName: \"kubernetes.io/projected/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-kube-api-access-fdhhl\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996553 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-cni-bin\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996564 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-socket-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996588 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-sys-fs\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996591 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-k8s-cni-cncf-io\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996611 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8a26bd-3e9b-4308-915d-f6fc9cd8394a-agent-certs\") pod \"konnectivity-agent-zhcw4\" (UID: \"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a\") " pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-kubernetes\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996685 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-os-release\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996714 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-kubelet\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysctl-d\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-registration-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996812 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-conf-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:40.054008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996835 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2115fb78-bf70-4dfc-8365-5caab1a40403-cni-binary-copy\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996834 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-socket-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996855 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-daemon-config\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996866 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-etc-kubernetes\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.996896 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996902 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-etc-kubernetes\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996907 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bede19a7-4918-4b48-8bf0-f5ade4d53795-host\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996946 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bede19a7-4918-4b48-8bf0-f5ade4d53795-host\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996948 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-kubernetes\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.996972 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:40.496951589 +0000 UTC m=+3.212170569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996988 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-var-lib-kubelet\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-netns\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997000 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-os-release\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996711 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-sys\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8a26bd-3e9b-4308-915d-f6fc9cd8394a-konnectivity-ca\") pod \"konnectivity-agent-zhcw4\" (UID: \"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a\") " pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:40.055110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997048 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd336a71-4bf2-4084-af55-e8a606870ddb-iptables-alerter-script\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-tmp\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997082 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysctl-d\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997124 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-cni-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997138 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-registration-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997148 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysctl-conf\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-cnibin\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65242094-074d-41d5-a8e9-d22a5c636b9f-dbus\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997205 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/bede19a7-4918-4b48-8bf0-f5ade4d53795-serviceca\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997281 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-hostroot\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997312 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-etc-selinux\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:40.057172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997358 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-var-lib-kubelet\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997385 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997415 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8qnm\" (UniqueName: \"kubernetes.io/projected/fd336a71-4bf2-4084-af55-e8a606870ddb-kube-api-access-f8qnm\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997437 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.996896 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/1aa4a323-253b-4c0e-907e-e8521c1c418f-sys-fs\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997496 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-systemd\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.997216 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997439 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-systemd\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997537 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-run\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:39.997575 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:40.497552736 +0000 UTC m=+3.212771692 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997617 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a8a26bd-3e9b-4308-915d-f6fc9cd8394a-konnectivity-ca\") pod \"konnectivity-agent-zhcw4\" (UID: \"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a\") " pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997636 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-socket-dir-parent\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997636 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-run\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.058134 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997678 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/fd336a71-4bf2-4084-af55-e8a606870ddb-iptables-alerter-script\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997687 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-netns\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997708 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/65242094-074d-41d5-a8e9-d22a5c636b9f-dbus\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd336a71-4bf2-4084-af55-e8a606870ddb-host-slash\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997725 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-var-lib-kubelet\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997735 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/03734ae6-5b46-440f-99a2-6eac216a3675-cnibin\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysconfig\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997755 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-hostroot\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997772 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-cni-binary-copy\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-multus-certs\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997845 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9zfj\" (UniqueName: \"kubernetes.io/projected/2115fb78-bf70-4dfc-8365-5caab1a40403-kube-api-access-f9zfj\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997868 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysconfig\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997817 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-socket-dir-parent\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997844 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-sysctl-conf\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997637 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-multus-cni-dir\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.060047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-tmp-dir\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997893 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2115fb78-bf70-4dfc-8365-5caab1a40403-host-run-multus-certs\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cndmb\" (UniqueName: \"kubernetes.io/projected/bede19a7-4918-4b48-8bf0-f5ade4d53795-kube-api-access-cndmb\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997775 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd336a71-4bf2-4084-af55-e8a606870ddb-host-slash\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997961 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-host\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.997985 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6rh\" (UniqueName: \"kubernetes.io/projected/03734ae6-5b46-440f-99a2-6eac216a3675-kube-api-access-nv6rh\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-hosts-file\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998036 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65242094-074d-41d5-a8e9-d22a5c636b9f-kubelet-config\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998110 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/65242094-074d-41d5-a8e9-d22a5c636b9f-kubelet-config\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998204 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-tmp-dir\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998290 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998334 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998344 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-host\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998336 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-hosts-file\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998400 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/03734ae6-5b46-440f-99a2-6eac216a3675-cni-binary-copy\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.061364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:39.998654 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.000372 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-tmp\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.000681 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a8a26bd-3e9b-4308-915d-f6fc9cd8394a-agent-certs\") pod \"konnectivity-agent-zhcw4\" (UID: \"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a\") " pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.009445 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5afa0111_b488_47f0_b10e_2c657d2f9057.slice/crio-5c8172fbf244409d8b75c7d8394078e6490491d55eecc2135c19c0d8e04b262a WatchSource:0}: Error finding container 5c8172fbf244409d8b75c7d8394078e6490491d55eecc2135c19c0d8e04b262a: Status 404 returned error can't find the container with id 5c8172fbf244409d8b75c7d8394078e6490491d55eecc2135c19c0d8e04b262a Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.017336 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.019725 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5x8s\" (UniqueName: \"kubernetes.io/projected/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-kube-api-access-g5x8s\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.019819 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdhhl\" (UniqueName: \"kubernetes.io/projected/b1d64b6b-9b5a-4eb1-88db-e29385e6faa7-kube-api-access-fdhhl\") pod \"node-resolver-nm6dl\" (UID: \"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7\") " pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.022154 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9zfj\" (UniqueName: \"kubernetes.io/projected/2115fb78-bf70-4dfc-8365-5caab1a40403-kube-api-access-f9zfj\") pod \"multus-p4mwp\" (UID: \"2115fb78-bf70-4dfc-8365-5caab1a40403\") " pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.022373 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6rh\" (UniqueName: \"kubernetes.io/projected/03734ae6-5b46-440f-99a2-6eac216a3675-kube-api-access-nv6rh\") pod \"multus-additional-cni-plugins-57tjt\" (UID: \"03734ae6-5b46-440f-99a2-6eac216a3675\") " pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.024759 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8qnm\" (UniqueName: \"kubernetes.io/projected/fd336a71-4bf2-4084-af55-e8a606870ddb-kube-api-access-f8qnm\") pod \"iptables-alerter-hhjqj\" (UID: \"fd336a71-4bf2-4084-af55-e8a606870ddb\") " pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.024853 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hhjqj" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.026955 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8a26bd_3e9b_4308_915d_f6fc9cd8394a.slice/crio-76675265b22ea29b04e97ab2ffca5dda01123c4c20a60f52a68a9c459e8abf7c WatchSource:0}: Error finding container 76675265b22ea29b04e97ab2ffca5dda01123c4c20a60f52a68a9c459e8abf7c: Status 404 returned error can't find the container with id 76675265b22ea29b04e97ab2ffca5dda01123c4c20a60f52a68a9c459e8abf7c Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.026990 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cndmb\" (UniqueName: \"kubernetes.io/projected/bede19a7-4918-4b48-8bf0-f5ade4d53795-kube-api-access-cndmb\") pod \"node-ca-796kr\" (UID: \"bede19a7-4918-4b48-8bf0-f5ade4d53795\") " pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.031683 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nm6dl" Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.032399 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.032420 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:23:40.064116 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.032435 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.032494 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:23:40.532474968 +0000 UTC m=+3.247693927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.034536 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ws2ss\" (UniqueName: \"kubernetes.io/projected/0f864ebe-e103-4a19-96c9-cd9d2beba99f-kube-api-access-ws2ss\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.034551 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd336a71_4bf2_4084_af55_e8a606870ddb.slice/crio-a689cd6bebc358043d1a5d062b47eaeaccf71911554db71a1afe60ac9a3a014e WatchSource:0}: Error finding container a689cd6bebc358043d1a5d062b47eaeaccf71911554db71a1afe60ac9a3a014e: Status 404 returned error can't find the container with id a689cd6bebc358043d1a5d062b47eaeaccf71911554db71a1afe60ac9a3a014e Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.036930 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5f6j\" (UniqueName: \"kubernetes.io/projected/1aa4a323-253b-4c0e-907e-e8521c1c418f-kube-api-access-g5f6j\") pod \"aws-ebs-csi-driver-node-hmz9j\" (UID: \"1aa4a323-253b-4c0e-907e-e8521c1c418f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.042538 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d64b6b_9b5a_4eb1_88db_e29385e6faa7.slice/crio-1a7c4c18bd478b1b6d210f11cadd6a72e5754093f4f89478ef17bec704e610ff WatchSource:0}: Error finding container 1a7c4c18bd478b1b6d210f11cadd6a72e5754093f4f89478ef17bec704e610ff: Status 404 returned error can't find the container with id 1a7c4c18bd478b1b6d210f11cadd6a72e5754093f4f89478ef17bec704e610ff Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.048309 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-57tjt" Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.054512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.058263 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03734ae6_5b46_440f_99a2_6eac216a3675.slice/crio-c61e43a84afa935411c01b6d17162d4ee5203c4d0bc1de1149a130f27f0ffb24 WatchSource:0}: Error finding container c61e43a84afa935411c01b6d17162d4ee5203c4d0bc1de1149a130f27f0ffb24: Status 404 returned error can't find the container with id c61e43a84afa935411c01b6d17162d4ee5203c4d0bc1de1149a130f27f0ffb24 Jul 06 16:23:40.065416 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.060380 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-796kr" Jul 06 16:23:40.076212 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.074233 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbede19a7_4918_4b48_8bf0_f5ade4d53795.slice/crio-1755ca8effc95de4219001c7cb88eee764ace382e40e51ee2d507374fd4fc7ea WatchSource:0}: Error finding container 1755ca8effc95de4219001c7cb88eee764ace382e40e51ee2d507374fd4fc7ea: Status 404 returned error can't find the container with id 1755ca8effc95de4219001c7cb88eee764ace382e40e51ee2d507374fd4fc7ea Jul 06 16:23:40.151390 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.080970 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ed8974ec-a8f4-4715-8ae0-a83357c84aa6-etc-tuned\") pod \"tuned-hmk6v\" (UID: \"ed8974ec-a8f4-4715-8ae0-a83357c84aa6\") " pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.309231 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.307632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-p4mwp" Jul 06 16:23:40.321465 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:23:40.321434 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2115fb78_bf70_4dfc_8365_5caab1a40403.slice/crio-dd060bb9d455c0de015fafd51933b5a1a781f0bf00235738944198608ba012b1 WatchSource:0}: Error finding container dd060bb9d455c0de015fafd51933b5a1a781f0bf00235738944198608ba012b1: Status 404 returned error can't find the container with id dd060bb9d455c0de015fafd51933b5a1a781f0bf00235738944198608ba012b1 Jul 06 16:23:40.339845 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.339821 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" Jul 06 16:23:40.502259 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.502217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:40.502467 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.502267 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:40.502467 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.502377 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:40.502467 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.502381 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:40.502467 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.502435 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:41.502416893 +0000 UTC m=+4.217635859 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:40.502857 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.502510 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:41.502489625 +0000 UTC m=+4.217708589 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:40.603844 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.603811 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:40.604003 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.603961 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:23:40.604003 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.603978 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:23:40.604003 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.603990 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:40.604161 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:40.604050 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:23:41.604032572 +0000 UTC m=+4.319251527 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:40.760065 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.759949 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:18:39 +0000 UTC" deadline="2028-03-24 00:36:59.261914932 +0000 UTC" Jul 06 16:23:40.760065 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.760018 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15032h13m18.501899342s" Jul 06 16:23:40.797522 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.797483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" event={"ID":"ed8974ec-a8f4-4715-8ae0-a83357c84aa6","Type":"ContainerStarted","Data":"879a89ae4c4e1af41cac96d43abe2ea88404d79d6b206e162cb7cbca02659047"} Jul 06 16:23:40.798847 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.798821 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p4mwp" event={"ID":"2115fb78-bf70-4dfc-8365-5caab1a40403","Type":"ContainerStarted","Data":"dd060bb9d455c0de015fafd51933b5a1a781f0bf00235738944198608ba012b1"} Jul 06 16:23:40.800610 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.800583 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-796kr" event={"ID":"bede19a7-4918-4b48-8bf0-f5ade4d53795","Type":"ContainerStarted","Data":"1755ca8effc95de4219001c7cb88eee764ace382e40e51ee2d507374fd4fc7ea"} Jul 06 16:23:40.801805 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.801774 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" event={"ID":"1aa4a323-253b-4c0e-907e-e8521c1c418f","Type":"ContainerStarted","Data":"32e3b5e8d4f67ecf07b8af3d05004e30a6055b973196f87218847abd99d8df35"} Jul 06 16:23:40.803079 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.803046 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerStarted","Data":"c61e43a84afa935411c01b6d17162d4ee5203c4d0bc1de1149a130f27f0ffb24"} Jul 06 16:23:40.804352 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.804328 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"5c8172fbf244409d8b75c7d8394078e6490491d55eecc2135c19c0d8e04b262a"} Jul 06 16:23:40.805684 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.805662 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nm6dl" event={"ID":"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7","Type":"ContainerStarted","Data":"1a7c4c18bd478b1b6d210f11cadd6a72e5754093f4f89478ef17bec704e610ff"} Jul 06 16:23:40.806979 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.806953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hhjqj" event={"ID":"fd336a71-4bf2-4084-af55-e8a606870ddb","Type":"ContainerStarted","Data":"a689cd6bebc358043d1a5d062b47eaeaccf71911554db71a1afe60ac9a3a014e"} Jul 06 16:23:40.808088 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:40.808041 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zhcw4" event={"ID":"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a","Type":"ContainerStarted","Data":"76675265b22ea29b04e97ab2ffca5dda01123c4c20a60f52a68a9c459e8abf7c"} Jul 06 16:23:41.511661 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.511619 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:41.512002 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.511677 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:41.512002 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.511854 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:41.512002 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.511917 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:43.511897461 +0000 UTC m=+6.227116420 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:41.512429 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.512314 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:41.512429 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.512366 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:43.512351469 +0000 UTC m=+6.227570424 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:41.612691 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.612631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:41.612896 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.612872 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:23:41.612896 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.612895 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:23:41.613085 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.612908 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:41.613085 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.612981 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:23:43.612961255 +0000 UTC m=+6.328180206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:41.760875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.760832 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-05 16:18:39 +0000 UTC" deadline="2028-01-02 11:51:10.558816058 +0000 UTC" Jul 06 16:23:41.760875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.760873 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13075h27m28.797948518s" Jul 06 16:23:41.789715 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.789635 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:41.789893 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.789753 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:41.790171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.790150 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:41.790262 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.790245 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:41.790331 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:41.790313 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:41.790441 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:41.790406 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:43.530580 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:43.530538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:43.531077 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:43.530595 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:43.531077 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.530763 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:43.531077 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.530840 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:47.530820175 +0000 UTC m=+10.246039130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:43.531256 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.531219 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:43.531313 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.531272 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:47.531254313 +0000 UTC m=+10.246473268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:43.632035 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:43.631993 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:43.632211 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.632192 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:23:43.632265 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.632220 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:23:43.632265 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.632235 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:43.632378 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.632293 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:23:47.632274599 +0000 UTC m=+10.347493556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:43.788970 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:43.788892 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:43.789161 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.789020 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:43.789427 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:43.789407 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:43.789527 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.789507 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:43.789641 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:43.789622 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:43.789767 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:43.789714 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:45.787649 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:45.787109 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:45.787649 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:45.787109 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:45.787649 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:45.787258 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:45.787649 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:45.787248 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:45.787649 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:45.787351 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:45.787649 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:45.787435 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:47.567492 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:47.567189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:47.567492 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:47.567241 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:47.567492 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.567367 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:47.567492 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.567370 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:47.567492 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.567435 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:55.567415761 +0000 UTC m=+18.282634714 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:47.567492 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.567453 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:23:55.567444437 +0000 UTC m=+18.282663392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:47.669222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:47.668627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:47.669222 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.668814 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:23:47.669222 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.668837 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:23:47.669222 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.668850 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:47.669222 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.668911 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:23:55.668891795 +0000 UTC m=+18.384110774 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:47.787897 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:47.787822 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:47.788049 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.787966 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:47.788049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:47.788007 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:47.788156 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.788094 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:47.788156 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:47.788129 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:47.788223 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:47.788190 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:49.788002 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:49.787970 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:49.788409 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:49.788086 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:49.788409 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:49.787970 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:49.788409 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:49.788111 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:49.788409 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:49.788218 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:49.788409 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:49.788312 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:51.787360 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:51.787322 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:51.787921 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:51.787367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:51.787921 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:51.787440 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:51.787921 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:51.787326 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:51.787921 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:51.787560 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:51.787921 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:51.787629 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:53.787335 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:53.787305 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:53.787335 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:53.787325 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:53.787855 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:53.787322 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:53.787855 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:53.787405 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:53.787855 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:53.787521 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:53.787855 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:53.787618 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:55.625474 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:55.625262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:55.625961 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.625435 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:55.625961 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:55.625559 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:55.625961 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.625610 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:24:11.625588187 +0000 UTC m=+34.340807143 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:23:55.625961 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.625672 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:55.625961 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.625726 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:24:11.625711667 +0000 UTC m=+34.340930621 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:23:55.726022 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:55.725991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:55.726174 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.726121 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:23:55.726174 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.726138 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:23:55.726174 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.726147 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:55.726306 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.726194 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:24:11.726181887 +0000 UTC m=+34.441400839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:23:55.787683 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:55.787646 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:55.787888 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:55.787651 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:55.787888 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.787777 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:55.787888 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.787858 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:55.788039 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:55.787666 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:55.788039 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:55.788014 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:57.789435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:57.789390 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:57.789881 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:57.789526 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:57.790112 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:57.790093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:57.790309 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:57.790289 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:57.790370 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:57.790351 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:57.791011 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:57.790477 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:59.786888 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.786865 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:23:59.787271 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.786866 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:23:59.787271 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:59.786985 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:23:59.787271 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.786873 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:23:59.787271 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:59.787046 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:23:59.787271 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:23:59.787133 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:23:59.845017 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.844981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" event={"ID":"a35c3367aea0364e635c7dba529cb3ff","Type":"ContainerStarted","Data":"b22591db77b7a7407c9270e862601403529868b952c26c20ab0b29102a88f761"} Jul 06 16:23:59.848760 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.848732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" event={"ID":"ed8974ec-a8f4-4715-8ae0-a83357c84aa6","Type":"ContainerStarted","Data":"7a354e889f4ec16af488183f769d046fcbebfe00e873cfae4f8a7e297529f15d"} Jul 06 16:23:59.851590 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.851559 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:23:59.852298 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.852104 2563 generic.go:356] "Generic (PLEG): container finished" podID="5afa0111-b488-47f0-b10e-2c657d2f9057" containerID="271e6a60d82a85df9b5724fce0af23ba1cb1a69acb9a4badd91977190fffa650" exitCode=1 Jul 06 16:23:59.852298 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.852146 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"70f072c2fb4fd98b0dfdeb2c1b78b77c9c4efc697e03b66b0e90f2b31ac79a29"} Jul 06 16:23:59.852298 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.852213 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerDied","Data":"271e6a60d82a85df9b5724fce0af23ba1cb1a69acb9a4badd91977190fffa650"} Jul 06 16:23:59.852298 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.852262 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"70a4494ed52fd52c72e65a5428395aa990ef226180b1f2dca2f5249b0c86b138"} Jul 06 16:23:59.871863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.871808 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-202.ec2.internal" podStartSLOduration=21.871761845 podStartE2EDuration="21.871761845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:23:59.870923618 +0000 UTC m=+22.586142592" watchObservedRunningTime="2026-07-06 16:23:59.871761845 +0000 UTC m=+22.586980821" Jul 06 16:23:59.892092 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:23:59.891884 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hmk6v" podStartSLOduration=3.911382274 podStartE2EDuration="22.891863686s" totalImagesPullingTime="18.980481412s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:23:59.891661664 +0000 UTC m=+22.606880666" watchObservedRunningTime="2026-07-06 16:23:59.891863686 +0000 UTC m=+22.607082661" Jul 06 16:24:00.855411 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.855173 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-p4mwp" event={"ID":"2115fb78-bf70-4dfc-8365-5caab1a40403","Type":"ContainerStarted","Data":"b514d286f45dd5a75ae4559359bf043abc912415c21de1fdd17d6a51c180f24c"} Jul 06 16:24:00.856645 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.856617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-796kr" event={"ID":"bede19a7-4918-4b48-8bf0-f5ade4d53795","Type":"ContainerStarted","Data":"974e675de436435f2257efa3a40bcf86028f7c0b3778d2a39e5edf9a05f7bce1"} Jul 06 16:24:00.857897 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.857877 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" event={"ID":"1aa4a323-253b-4c0e-907e-e8521c1c418f","Type":"ContainerStarted","Data":"ada37e05b6253aa6578d3311248db254d18f26950ac7a54a904616268722f9f5"} Jul 06 16:24:00.859192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.859171 2563 generic.go:356] "Generic (PLEG): container finished" podID="03734ae6-5b46-440f-99a2-6eac216a3675" containerID="48e5f489516b96c1fb39841b77df8e3d5b553dd3154d35e52e65ead8480df5e1" exitCode=0 Jul 06 16:24:00.859284 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.859231 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerDied","Data":"48e5f489516b96c1fb39841b77df8e3d5b553dd3154d35e52e65ead8480df5e1"} Jul 06 16:24:00.862060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.862041 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:24:00.862399 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.862382 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"a75f84c8976e7b1dadccc5ba875a7c03e76f247c9eb41abdf531b8d0cce2be45"} Jul 06 16:24:00.862461 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.862447 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"b79fe49b86d31cdfbf84803c6edb31fe3f2ef50c37ec4189dd2530ed8b8be1c0"} Jul 06 16:24:00.862513 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.862500 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"824241c1310d1a7edcc6615c08e5bd66a3bcb2efe2d0caf687112660365d43e2"} Jul 06 16:24:00.863806 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.863777 2563 generic.go:356] "Generic (PLEG): container finished" podID="897d6bcf4199302dfb88bc40b83b5bcb" containerID="a30a1defa389dca38b6446068c0865c9fe3c44c3fdf82b8adf0b9923c98ba86e" exitCode=0 Jul 06 16:24:00.863877 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.863824 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" event={"ID":"897d6bcf4199302dfb88bc40b83b5bcb","Type":"ContainerDied","Data":"a30a1defa389dca38b6446068c0865c9fe3c44c3fdf82b8adf0b9923c98ba86e"} Jul 06 16:24:00.865330 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.865287 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nm6dl" event={"ID":"b1d64b6b-9b5a-4eb1-88db-e29385e6faa7","Type":"ContainerStarted","Data":"efa9c5f86a7031ac59a7805c393a5a61d5e4907b5706e9b36248245e2cfc0489"} Jul 06 16:24:00.866677 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.866659 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hhjqj" event={"ID":"fd336a71-4bf2-4084-af55-e8a606870ddb","Type":"ContainerStarted","Data":"5b6ca23c65e93621be06aab8ecc6ec670d06964ef5d4fd30e31801025bb82ab5"} Jul 06 16:24:00.867863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.867837 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zhcw4" event={"ID":"0a8a26bd-3e9b-4308-915d-f6fc9cd8394a","Type":"ContainerStarted","Data":"f9639bbe1b565dc08f9e034ea18ad8cabbe65303d8792598666e79e380e986aa"} Jul 06 16:24:00.899472 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.899403 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-p4mwp" podStartSLOduration=4.368926055 podStartE2EDuration="23.899382481s" totalImagesPullingTime="19.530456426s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:00.899265324 +0000 UTC m=+23.614484299" watchObservedRunningTime="2026-07-06 16:24:00.899382481 +0000 UTC m=+23.614601457" Jul 06 16:24:00.964638 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.964566 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-796kr" podStartSLOduration=3.708878006 podStartE2EDuration="22.964549997s" totalImagesPullingTime="19.255671991s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:00.931786347 +0000 UTC m=+23.647005319" watchObservedRunningTime="2026-07-06 16:24:00.964549997 +0000 UTC m=+23.679768971" Jul 06 16:24:00.964846 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:00.964823 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zhcw4" podStartSLOduration=13.243427655 podStartE2EDuration="23.96481567s" totalImagesPullingTime="10.721388015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:00.964214869 +0000 UTC m=+23.679433856" watchObservedRunningTime="2026-07-06 16:24:00.96481567 +0000 UTC m=+23.680034638" Jul 06 16:24:01.047314 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.047261 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nm6dl" podStartSLOduration=4.780105297 podStartE2EDuration="24.047243615s" totalImagesPullingTime="19.267138318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:01.012550338 +0000 UTC m=+23.727769313" watchObservedRunningTime="2026-07-06 16:24:01.047243615 +0000 UTC m=+23.762462589" Jul 06 16:24:01.120726 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.120664 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hhjqj" podStartSLOduration=5.153025456 podStartE2EDuration="24.120644541s" totalImagesPullingTime="18.967619085s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:01.119878438 +0000 UTC m=+23.835097436" watchObservedRunningTime="2026-07-06 16:24:01.120644541 +0000 UTC m=+23.835863516" Jul 06 16:24:01.787239 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.787213 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:01.787421 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.787219 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:01.787421 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.787333 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:01.787421 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:01.787331 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:24:01.787588 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:01.787535 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:24:01.787656 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:01.787633 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:24:01.808979 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.808954 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 06 16:24:01.873547 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.873513 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" event={"ID":"897d6bcf4199302dfb88bc40b83b5bcb","Type":"ContainerStarted","Data":"80be01dd156d3f2ce46cafec888a449c0d147516d2895909bad3fc6750cf392d"} Jul 06 16:24:01.875270 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:01.875245 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" event={"ID":"1aa4a323-253b-4c0e-907e-e8521c1c418f","Type":"ContainerStarted","Data":"ec6e07c0f52f3be94fcf22f860c21102aba47863e2933d41d03ccb56a0617c71"} Jul 06 16:24:02.804913 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:02.804813 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-06T16:24:01.808974296Z","UUID":"05d46dbb-f38a-4f78-9e0e-2f29ff92c25b","Handler":null,"Name":"","Endpoint":""} Jul 06 16:24:02.806235 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:02.806218 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 06 16:24:02.806335 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:02.806253 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 06 16:24:02.881340 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:02.881313 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:24:02.881850 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:02.881818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"5b8815e9de683f19aaf9b13a3666c71b9e3d873728bd220bb178d6d9f229d1fe"} Jul 06 16:24:03.786957 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:03.786921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:03.787154 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:03.786921 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:03.787154 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:03.787027 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:24:03.787154 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:03.786927 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:03.787154 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:03.787112 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:24:03.787316 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:03.787194 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:24:03.886140 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:03.886104 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" event={"ID":"1aa4a323-253b-4c0e-907e-e8521c1c418f","Type":"ContainerStarted","Data":"0494d123d297cab9652d33cc11f02387c7374f044426e2ca26140a75f3ffd064"} Jul 06 16:24:03.909272 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:03.909223 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hmz9j" podStartSLOduration=4.241141087 podStartE2EDuration="25.909206954s" totalImagesPullingTime="21.668065867s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:03.907538481 +0000 UTC m=+26.622757455" watchObservedRunningTime="2026-07-06 16:24:03.909206954 +0000 UTC m=+26.624425925" Jul 06 16:24:03.909389 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:03.909320 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-202.ec2.internal" podStartSLOduration=25.90931483 podStartE2EDuration="25.90931483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:01.894736316 +0000 UTC m=+24.609955290" watchObservedRunningTime="2026-07-06 16:24:03.90931483 +0000 UTC m=+26.624533804" Jul 06 16:24:04.696275 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.696097 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:24:04.696861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.696844 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:24:04.891664 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.891632 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:24:04.892419 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.892017 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"ad4031c0671275cdff4bcfe9b1dada63388c782982ca767b31e9fe18e6c33315"} Jul 06 16:24:04.892497 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.892484 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:24:04.892599 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.892581 2563 scope.go:117] "RemoveContainer" containerID="271e6a60d82a85df9b5724fce0af23ba1cb1a69acb9a4badd91977190fffa650" Jul 06 16:24:04.892648 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:04.892614 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zhcw4" Jul 06 16:24:05.787652 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.787620 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:05.787813 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.787621 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:05.787813 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.787735 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:05.787813 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:05.787739 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:24:05.788024 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:05.787837 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:24:05.788024 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:05.787942 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:24:05.897956 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.897929 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:24:05.898539 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.898344 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" event={"ID":"5afa0111-b488-47f0-b10e-2c657d2f9057","Type":"ContainerStarted","Data":"cc17f0f76b013ce39c0bc3848c4ea2dd929a47506b0dbff2839560a7f485c857"} Jul 06 16:24:05.898600 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.898586 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:24:05.898689 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.898674 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:24:05.898740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.898725 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:24:05.914121 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.914096 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:24:05.914531 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.914516 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:24:05.935780 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:05.935721 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" podStartSLOduration=9.539408659 podStartE2EDuration="28.935704305s" totalImagesPullingTime="19.396295646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:05.935187234 +0000 UTC m=+28.650406233" watchObservedRunningTime="2026-07-06 16:24:05.935704305 +0000 UTC m=+28.650923278" Jul 06 16:24:06.163492 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:06.163462 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m285v"] Jul 06 16:24:06.163696 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:06.163564 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:06.163775 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:06.163680 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:24:06.167172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:06.167142 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxtrg"] Jul 06 16:24:06.167329 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:06.167246 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:06.167433 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:06.167408 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:24:06.167989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:06.167967 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d6nms"] Jul 06 16:24:06.171773 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:06.168585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:06.171773 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:06.168828 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:24:07.788312 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:07.788112 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:07.788722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:07.788178 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:07.788722 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:07.788385 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:24:07.788722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:07.788198 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:07.788722 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:07.788463 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:24:07.788722 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:07.788563 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:24:08.906598 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:08.906557 2563 generic.go:356] "Generic (PLEG): container finished" podID="03734ae6-5b46-440f-99a2-6eac216a3675" containerID="4d36580831ac49e0a461b8b5c36a2e5364bfcfcd4c3de0571f2287e0e37e8a63" exitCode=0 Jul 06 16:24:08.906989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:08.906622 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerDied","Data":"4d36580831ac49e0a461b8b5c36a2e5364bfcfcd4c3de0571f2287e0e37e8a63"} Jul 06 16:24:09.787846 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:09.787811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:09.788013 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:09.787931 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xxtrg" podUID="6b8a8d6b-f434-4396-b90e-aae39e897dab" Jul 06 16:24:09.788013 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:09.787952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:09.788013 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:09.787811 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:09.788176 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:09.788044 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-m285v" podUID="65242094-074d-41d5-a8e9-d22a5c636b9f" Jul 06 16:24:09.788229 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:09.788169 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-d6nms" podUID="0f864ebe-e103-4a19-96c9-cd9d2beba99f" Jul 06 16:24:10.923047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:10.922862 2563 generic.go:356] "Generic (PLEG): container finished" podID="03734ae6-5b46-440f-99a2-6eac216a3675" containerID="8d70b5c6fc889334474e770acafc8c14b49164cfa6f0f3dcb50bfcd215623237" exitCode=0 Jul 06 16:24:10.923454 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:10.922944 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerDied","Data":"8d70b5c6fc889334474e770acafc8c14b49164cfa6f0f3dcb50bfcd215623237"} Jul 06 16:24:11.473716 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.473689 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-202.ec2.internal" event="NodeReady" Jul 06 16:24:11.473878 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.473825 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 06 16:24:11.523479 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.523447 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-fc8cdfbd7-8k66g"] Jul 06 16:24:11.545094 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.545008 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qtdqx"] Jul 06 16:24:11.545248 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.545171 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.549510 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.549254 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 06 16:24:11.549669 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.549594 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 06 16:24:11.550658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.550635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qh9j6\"" Jul 06 16:24:11.552770 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.552747 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 06 16:24:11.559950 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.559928 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 06 16:24:11.567370 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.567351 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk"] Jul 06 16:24:11.567480 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.567467 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.574002 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.573977 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 06 16:24:11.574674 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.574656 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wmlq9\"" Jul 06 16:24:11.574932 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.574917 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 06 16:24:11.587659 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.587636 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-996dbbc49-g98f5"] Jul 06 16:24:11.587756 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.587742 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.591737 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.591717 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 06 16:24:11.591860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.591845 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 06 16:24:11.591907 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.591894 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9stxg\"" Jul 06 16:24:11.594097 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.594078 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 06 16:24:11.594220 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.594204 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 06 16:24:11.609558 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.609537 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-nrkt5"] Jul 06 16:24:11.609665 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.609646 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.613714 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.613686 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 06 16:24:11.613714 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.613711 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-68jqv\"" Jul 06 16:24:11.613886 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.613827 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 06 16:24:11.613989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.613970 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 06 16:24:11.614132 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.614118 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 06 16:24:11.614249 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.614133 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 06 16:24:11.614499 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.614485 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 06 16:24:11.631692 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.631662 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5p5zj"] Jul 06 16:24:11.631854 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.631785 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.635376 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.635356 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 06 16:24:11.635522 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.635443 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 06 16:24:11.636016 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.635997 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 06 16:24:11.636107 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.636020 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 06 16:24:11.636107 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.636060 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rpfpg\"" Jul 06 16:24:11.637668 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637649 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-image-registry-private-configuration\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v8lh\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-kube-api-access-2v8lh\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637715 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-certificates\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877b735-824f-46f9-b2e4-016dc8bd2725-tmp-dir\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637834 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637873 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8269cb04-9e02-4190-b15b-f64a1fc59bbc-ca-trust-extracted\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637899 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-installation-pull-secrets\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637924 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-bound-sa-token\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637959 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.637988 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.637987 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.637973 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 06 16:24:11.638366 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.638037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lj4s\" (UniqueName: \"kubernetes.io/projected/a877b735-824f-46f9-b2e4-016dc8bd2725-kube-api-access-9lj4s\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.638366 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.638074 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret podName:65242094-074d-41d5-a8e9-d22a5c636b9f nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.638031813 +0000 UTC m=+66.353250779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret") pod "global-pull-secret-syncer-m285v" (UID: "65242094-074d-41d5-a8e9-d22a5c636b9f") : object "kube-system"/"original-pull-secret" not registered Jul 06 16:24:11.638366 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.638143 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:24:11.638366 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.638206 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs podName:0f864ebe-e103-4a19-96c9-cd9d2beba99f nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.638193154 +0000 UTC m=+66.353412107 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs") pod "network-metrics-daemon-d6nms" (UID: "0f864ebe-e103-4a19-96c9-cd9d2beba99f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 06 16:24:11.638366 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.638222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-trusted-ca\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.638366 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.638260 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a877b735-824f-46f9-b2e4-016dc8bd2725-config-volume\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.644116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.644097 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 06 16:24:11.653199 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653179 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc8cdfbd7-8k66g"] Jul 06 16:24:11.653262 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653207 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qtdqx"] Jul 06 16:24:11.653262 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653224 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5p5zj"] Jul 06 16:24:11.653262 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653248 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk"] Jul 06 16:24:11.653359 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653316 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.653399 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653361 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-996dbbc49-g98f5"] Jul 06 16:24:11.653399 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.653385 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-nrkt5"] Jul 06 16:24:11.657552 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.657525 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 06 16:24:11.657661 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.657530 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 06 16:24:11.657717 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.657672 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 06 16:24:11.657863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.657846 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-25tl8\"" Jul 06 16:24:11.739036 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.738998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcvs\" (UniqueName: \"kubernetes.io/projected/73c3b173-35b2-41ac-bd6f-e568f1229742-kube-api-access-2bcvs\") pod \"ingress-canary-5p5zj\" (UID: \"73c3b173-35b2-41ac-bd6f-e568f1229742\") " pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.739036 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d32c3f81-2740-490c-b120-f67350df8f95-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.739215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739058 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d32c3f81-2740-490c-b120-f67350df8f95-tmp\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.739215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739122 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:11.739295 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.739239 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 06 16:24:11.739295 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.739258 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 06 16:24:11.739295 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739254 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-trusted-ca\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.739295 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.739270 2563 projected.go:196] Error preparing data for projected volume kube-api-access-5v5zn for pod openshift-network-diagnostics/network-check-target-xxtrg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:24:11.739295 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-stats-auth\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.739429 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a877b735-824f-46f9-b2e4-016dc8bd2725-config-volume\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.739429 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.739338 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn podName:6b8a8d6b-f434-4396-b90e-aae39e897dab nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.739321017 +0000 UTC m=+66.454539969 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-5v5zn" (UniqueName: "kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn") pod "network-check-target-xxtrg" (UID: "6b8a8d6b-f434-4396-b90e-aae39e897dab") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 06 16:24:11.739429 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739367 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d32c3f81-2740-490c-b120-f67350df8f95-snapshots\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.739429 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739400 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9254\" (UniqueName: \"kubernetes.io/projected/d32c3f81-2740-490c-b120-f67350df8f95-kube-api-access-z9254\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.739584 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-image-registry-private-configuration\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.739584 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739455 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-default-certificate\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.739584 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739481 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-certificates\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.739584 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739510 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d32c3f81-2740-490c-b120-f67350df8f95-serving-cert\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.739774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739590 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-installation-pull-secrets\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.739774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739632 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-bound-sa-token\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.739774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4r68\" (UniqueName: \"kubernetes.io/projected/d1148015-927a-46cb-8fa4-e4520fc74763-kube-api-access-p4r68\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.739774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739697 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba335293-eebb-4b84-87aa-84258e6c4111-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.739774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.739774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739757 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lj4s\" (UniqueName: \"kubernetes.io/projected/a877b735-824f-46f9-b2e4-016dc8bd2725-kube-api-access-9lj4s\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739829 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49wr2\" (UniqueName: \"kubernetes.io/projected/ba335293-eebb-4b84-87aa-84258e6c4111-kube-api-access-49wr2\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739853 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c3b173-35b2-41ac-bd6f-e568f1229742-cert\") pod \"ingress-canary-5p5zj\" (UID: \"73c3b173-35b2-41ac-bd6f-e568f1229742\") " pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739911 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v8lh\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-kube-api-access-2v8lh\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739936 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739967 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.739992 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d32c3f81-2740-490c-b120-f67350df8f95-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740011 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a877b735-824f-46f9-b2e4-016dc8bd2725-config-volume\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740018 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877b735-824f-46f9-b2e4-016dc8bd2725-tmp-dir\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.740075 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740074 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8269cb04-9e02-4190-b15b-f64a1fc59bbc-ca-trust-extracted\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740099 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-certificates\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.740191 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.740257 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls podName:a877b735-824f-46f9-b2e4-016dc8bd2725 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:12.24024144 +0000 UTC m=+34.955460397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls") pod "dns-default-qtdqx" (UID: "a877b735-824f-46f9-b2e4-016dc8bd2725") : secret "dns-default-metrics-tls" not found Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.740344 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.740357 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc8cdfbd7-8k66g: secret "image-registry-tls" not found Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.740391 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls podName:8269cb04-9e02-4190-b15b-f64a1fc59bbc nodeName:}" failed. No retries permitted until 2026-07-06 16:24:12.240379514 +0000 UTC m=+34.955598467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls") pod "image-registry-fc8cdfbd7-8k66g" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc") : secret "image-registry-tls" not found Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740403 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a877b735-824f-46f9-b2e4-016dc8bd2725-tmp-dir\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740405 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-trusted-ca\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.740520 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.740500 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8269cb04-9e02-4190-b15b-f64a1fc59bbc-ca-trust-extracted\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.743606 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.743575 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-image-registry-private-configuration\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.743721 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.743605 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-installation-pull-secrets\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.751456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.751428 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-bound-sa-token\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.751948 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.751927 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lj4s\" (UniqueName: \"kubernetes.io/projected/a877b735-824f-46f9-b2e4-016dc8bd2725-kube-api-access-9lj4s\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:11.752595 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.752572 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v8lh\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-kube-api-access-2v8lh\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:11.787709 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.787680 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:11.787908 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.787686 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:11.787908 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.787787 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:11.791005 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.790975 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 16:24:11.791005 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.790979 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 16:24:11.791184 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.790979 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 16:24:11.791184 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.791097 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6db6n\"" Jul 06 16:24:11.791184 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.791141 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hn2l7\"" Jul 06 16:24:11.791394 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.791374 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 16:24:11.841343 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841312 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.841499 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841355 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.841499 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841396 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.841616 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.841511 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:11.841616 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.841569 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:12.341549016 +0000 UTC m=+35.056767972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : configmap references non-existent config key: service-ca.crt Jul 06 16:24:11.841616 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.841593 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls podName:ba335293-eebb-4b84-87aa-84258e6c4111 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:12.341585469 +0000 UTC m=+35.056804426 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jk7mk" (UID: "ba335293-eebb-4b84-87aa-84258e6c4111") : secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:11.841616 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.841616 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841618 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d32c3f81-2740-490c-b120-f67350df8f95-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:11.841656 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:12.341638966 +0000 UTC m=+35.056857918 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : secret "router-metrics-certs-default" not found Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcvs\" (UniqueName: \"kubernetes.io/projected/73c3b173-35b2-41ac-bd6f-e568f1229742-kube-api-access-2bcvs\") pod \"ingress-canary-5p5zj\" (UID: \"73c3b173-35b2-41ac-bd6f-e568f1229742\") " pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841724 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d32c3f81-2740-490c-b120-f67350df8f95-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d32c3f81-2740-490c-b120-f67350df8f95-tmp\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841778 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-stats-auth\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.841841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841835 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d32c3f81-2740-490c-b120-f67350df8f95-snapshots\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841869 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9254\" (UniqueName: \"kubernetes.io/projected/d32c3f81-2740-490c-b120-f67350df8f95-kube-api-access-z9254\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-default-certificate\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841931 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d32c3f81-2740-490c-b120-f67350df8f95-serving-cert\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.841971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4r68\" (UniqueName: \"kubernetes.io/projected/d1148015-927a-46cb-8fa4-e4520fc74763-kube-api-access-p4r68\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.842003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba335293-eebb-4b84-87aa-84258e6c4111-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.842059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49wr2\" (UniqueName: \"kubernetes.io/projected/ba335293-eebb-4b84-87aa-84258e6c4111-kube-api-access-49wr2\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.842192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.842082 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c3b173-35b2-41ac-bd6f-e568f1229742-cert\") pod \"ingress-canary-5p5zj\" (UID: \"73c3b173-35b2-41ac-bd6f-e568f1229742\") " pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.842526 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.842497 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d32c3f81-2740-490c-b120-f67350df8f95-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.842723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.842697 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d32c3f81-2740-490c-b120-f67350df8f95-tmp\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.843365 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.843045 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d32c3f81-2740-490c-b120-f67350df8f95-snapshots\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.843685 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.843358 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/ba335293-eebb-4b84-87aa-84258e6c4111-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.843685 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.843403 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d32c3f81-2740-490c-b120-f67350df8f95-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.845183 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.845158 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-stats-auth\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.845308 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.845290 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-default-certificate\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.845374 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.845329 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/73c3b173-35b2-41ac-bd6f-e568f1229742-cert\") pod \"ingress-canary-5p5zj\" (UID: \"73c3b173-35b2-41ac-bd6f-e568f1229742\") " pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.845562 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.845544 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d32c3f81-2740-490c-b120-f67350df8f95-serving-cert\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.856228 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.856144 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9254\" (UniqueName: \"kubernetes.io/projected/d32c3f81-2740-490c-b120-f67350df8f95-kube-api-access-z9254\") pod \"insights-operator-5c847dcfc4-nrkt5\" (UID: \"d32c3f81-2740-490c-b120-f67350df8f95\") " pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.856411 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.856379 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4r68\" (UniqueName: \"kubernetes.io/projected/d1148015-927a-46cb-8fa4-e4520fc74763-kube-api-access-p4r68\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:11.856670 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.856655 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49wr2\" (UniqueName: \"kubernetes.io/projected/ba335293-eebb-4b84-87aa-84258e6c4111-kube-api-access-49wr2\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:11.857246 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.857228 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcvs\" (UniqueName: \"kubernetes.io/projected/73c3b173-35b2-41ac-bd6f-e568f1229742-kube-api-access-2bcvs\") pod \"ingress-canary-5p5zj\" (UID: \"73c3b173-35b2-41ac-bd6f-e568f1229742\") " pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:11.940974 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.940944 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" Jul 06 16:24:11.963221 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:11.963191 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5p5zj" Jul 06 16:24:12.165027 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.164946 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-nrkt5"] Jul 06 16:24:12.166460 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.166440 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5p5zj"] Jul 06 16:24:12.175892 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:12.175861 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32c3f81_2740_490c_b120_f67350df8f95.slice/crio-b0d7d657efeb76b8e5d45c60c35b1889c89b3a6795bfda1b3932d946aacad7b2 WatchSource:0}: Error finding container b0d7d657efeb76b8e5d45c60c35b1889c89b3a6795bfda1b3932d946aacad7b2: Status 404 returned error can't find the container with id b0d7d657efeb76b8e5d45c60c35b1889c89b3a6795bfda1b3932d946aacad7b2 Jul 06 16:24:12.176107 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:12.176081 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c3b173_35b2_41ac_bd6f_e568f1229742.slice/crio-129b82566edd5ed038936f6fb09cbe05105cc729af8e27ffc45e712680b8a95e WatchSource:0}: Error finding container 129b82566edd5ed038936f6fb09cbe05105cc729af8e27ffc45e712680b8a95e: Status 404 returned error can't find the container with id 129b82566edd5ed038936f6fb09cbe05105cc729af8e27ffc45e712680b8a95e Jul 06 16:24:12.246601 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.246569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:12.246752 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.246668 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:12.246752 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.246718 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:24:12.246752 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.246736 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc8cdfbd7-8k66g: secret "image-registry-tls" not found Jul 06 16:24:12.246903 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.246761 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:24:12.246903 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.246786 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls podName:8269cb04-9e02-4190-b15b-f64a1fc59bbc nodeName:}" failed. No retries permitted until 2026-07-06 16:24:13.246773138 +0000 UTC m=+35.961992090 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls") pod "image-registry-fc8cdfbd7-8k66g" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc") : secret "image-registry-tls" not found Jul 06 16:24:12.246903 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.246824 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls podName:a877b735-824f-46f9-b2e4-016dc8bd2725 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:13.246812286 +0000 UTC m=+35.962031252 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls") pod "dns-default-qtdqx" (UID: "a877b735-824f-46f9-b2e4-016dc8bd2725") : secret "dns-default-metrics-tls" not found Jul 06 16:24:12.348200 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.348160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:12.348379 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.348224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:12.348379 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.348258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:12.348484 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.348415 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:12.348484 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.348475 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls podName:ba335293-eebb-4b84-87aa-84258e6c4111 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:13.348458253 +0000 UTC m=+36.063677205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jk7mk" (UID: "ba335293-eebb-4b84-87aa-84258e6c4111") : secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:12.348597 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.348483 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 16:24:12.348597 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.348541 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:13.34852138 +0000 UTC m=+36.063740348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : configmap references non-existent config key: service-ca.crt Jul 06 16:24:12.348597 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:12.348569 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:13.348556141 +0000 UTC m=+36.063775108 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : secret "router-metrics-certs-default" not found Jul 06 16:24:12.929148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.928959 2563 generic.go:356] "Generic (PLEG): container finished" podID="03734ae6-5b46-440f-99a2-6eac216a3675" containerID="79be82a05b27b2b7f6bb3b1c0cfdc3f7dc99fa0daa3170973757fe95db202c01" exitCode=0 Jul 06 16:24:12.929410 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.929035 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerDied","Data":"79be82a05b27b2b7f6bb3b1c0cfdc3f7dc99fa0daa3170973757fe95db202c01"} Jul 06 16:24:12.930371 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.930348 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5p5zj" event={"ID":"73c3b173-35b2-41ac-bd6f-e568f1229742","Type":"ContainerStarted","Data":"129b82566edd5ed038936f6fb09cbe05105cc729af8e27ffc45e712680b8a95e"} Jul 06 16:24:12.931324 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:12.931305 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" event={"ID":"d32c3f81-2740-490c-b120-f67350df8f95","Type":"ContainerStarted","Data":"b0d7d657efeb76b8e5d45c60c35b1889c89b3a6795bfda1b3932d946aacad7b2"} Jul 06 16:24:13.256113 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:13.256029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:13.256880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.256187 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:24:13.256880 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:13.256197 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:13.256880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.256252 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls podName:a877b735-824f-46f9-b2e4-016dc8bd2725 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:15.256230722 +0000 UTC m=+37.971449674 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls") pod "dns-default-qtdqx" (UID: "a877b735-824f-46f9-b2e4-016dc8bd2725") : secret "dns-default-metrics-tls" not found Jul 06 16:24:13.256880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.256306 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:24:13.256880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.256321 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc8cdfbd7-8k66g: secret "image-registry-tls" not found Jul 06 16:24:13.256880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.256366 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls podName:8269cb04-9e02-4190-b15b-f64a1fc59bbc nodeName:}" failed. No retries permitted until 2026-07-06 16:24:15.256353647 +0000 UTC m=+37.971572603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls") pod "image-registry-fc8cdfbd7-8k66g" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc") : secret "image-registry-tls" not found Jul 06 16:24:13.356998 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:13.356956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:13.357215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:13.357021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:13.357215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:13.357049 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:13.357215 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.357104 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:15.357083259 +0000 UTC m=+38.072302214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : configmap references non-existent config key: service-ca.crt Jul 06 16:24:13.357215 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.357146 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:13.357215 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.357174 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 16:24:13.357475 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.357232 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:15.357211345 +0000 UTC m=+38.072430319 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : secret "router-metrics-certs-default" not found Jul 06 16:24:13.357475 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:13.357272 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls podName:ba335293-eebb-4b84-87aa-84258e6c4111 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:15.357261129 +0000 UTC m=+38.072480091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jk7mk" (UID: "ba335293-eebb-4b84-87aa-84258e6c4111") : secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:15.276395 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:15.276354 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:15.276880 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:15.276441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:15.276880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.276573 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:24:15.276880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.276633 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:24:15.276880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.276657 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc8cdfbd7-8k66g: secret "image-registry-tls" not found Jul 06 16:24:15.276880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.276644 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls podName:a877b735-824f-46f9-b2e4-016dc8bd2725 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.276624436 +0000 UTC m=+41.991843403 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls") pod "dns-default-qtdqx" (UID: "a877b735-824f-46f9-b2e4-016dc8bd2725") : secret "dns-default-metrics-tls" not found Jul 06 16:24:15.276880 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.276728 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls podName:8269cb04-9e02-4190-b15b-f64a1fc59bbc nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.2767093 +0000 UTC m=+41.991928257 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls") pod "image-registry-fc8cdfbd7-8k66g" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc") : secret "image-registry-tls" not found Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:15.377472 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:15.377529 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:15.377562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.377634 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.377652 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.377631701 +0000 UTC m=+42.092850662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : configmap references non-existent config key: service-ca.crt Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.377683 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.37767397 +0000 UTC m=+42.092892922 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : secret "router-metrics-certs-default" not found Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.377709 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:15.377845 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:15.377763 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls podName:ba335293-eebb-4b84-87aa-84258e6c4111 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.377746138 +0000 UTC m=+42.092965105 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jk7mk" (UID: "ba335293-eebb-4b84-87aa-84258e6c4111") : secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:16.942811 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:16.942631 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5p5zj" event={"ID":"73c3b173-35b2-41ac-bd6f-e568f1229742","Type":"ContainerStarted","Data":"00e2cb4194e70eff2dce9ae818f885374a946f63cf9a50a7c8128fb1ed2ee8e2"} Jul 06 16:24:16.944521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:16.944492 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" event={"ID":"d32c3f81-2740-490c-b120-f67350df8f95","Type":"ContainerStarted","Data":"b234d1b0f44603dc79269c064dccf7b81505e24e12166cf376c566a3b9b1a8e6"} Jul 06 16:24:16.966620 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:16.966561 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5p5zj" podStartSLOduration=2.085878627 podStartE2EDuration="5.966541725s" totalImagesPullingTime="3.880663098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:16.963577841 +0000 UTC m=+39.678796816" watchObservedRunningTime="2026-07-06 16:24:16.966541725 +0000 UTC m=+39.681760700" Jul 06 16:24:17.001946 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.001891 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" podStartSLOduration=6.12921521 podStartE2EDuration="10.001872974s" totalImagesPullingTime="3.872657764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:17.001185195 +0000 UTC m=+39.716404171" watchObservedRunningTime="2026-07-06 16:24:17.001872974 +0000 UTC m=+39.717091947" Jul 06 16:24:17.189494 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.189465 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sdkwl"] Jul 06 16:24:17.222869 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.222563 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdkwl"] Jul 06 16:24:17.222869 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.222735 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.228838 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.228811 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 06 16:24:17.228994 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.228893 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 06 16:24:17.228994 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.228849 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gb7wf\"" Jul 06 16:24:17.398887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.398849 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m77km\" (UniqueName: \"kubernetes.io/projected/3238e000-945c-4464-aa32-2f2e0843d959-kube-api-access-m77km\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.399061 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.398908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.399061 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.399018 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.399167 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.399086 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3238e000-945c-4464-aa32-2f2e0843d959-data-volume\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.399167 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.399133 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3238e000-945c-4464-aa32-2f2e0843d959-crio-socket\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500408 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500408 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500369 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3238e000-945c-4464-aa32-2f2e0843d959-data-volume\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500639 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500422 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3238e000-945c-4464-aa32-2f2e0843d959-crio-socket\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500639 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:17.500444 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:24:17.500639 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:17.500525 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:18.000502874 +0000 UTC m=+40.715721826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : configmap "kube-rbac-proxy" not found Jul 06 16:24:17.500639 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500573 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m77km\" (UniqueName: \"kubernetes.io/projected/3238e000-945c-4464-aa32-2f2e0843d959-kube-api-access-m77km\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500639 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500607 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500909 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500711 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3238e000-945c-4464-aa32-2f2e0843d959-crio-socket\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.500909 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:17.500731 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:24:17.500909 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:17.500787 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:18.000770041 +0000 UTC m=+40.715988999 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : secret "insights-runtime-extractor-tls" not found Jul 06 16:24:17.501058 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.500995 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3238e000-945c-4464-aa32-2f2e0843d959-data-volume\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:17.523775 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:17.523740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m77km\" (UniqueName: \"kubernetes.io/projected/3238e000-945c-4464-aa32-2f2e0843d959-kube-api-access-m77km\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:18.005764 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:18.005729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:18.006229 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:18.005783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:18.006229 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:18.005936 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:24:18.006229 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:18.005976 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 06 16:24:18.006229 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:18.006031 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.005998434 +0000 UTC m=+41.721217405 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : secret "insights-runtime-extractor-tls" not found Jul 06 16:24:18.006229 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:18.006053 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:19.006042047 +0000 UTC m=+41.721261002 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : configmap "kube-rbac-proxy" not found Jul 06 16:24:18.739604 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:18.739578 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nm6dl_b1d64b6b-9b5a-4eb1-88db-e29385e6faa7/dns-node-resolver/0.log" Jul 06 16:24:19.017293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.017213 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:19.017293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.017261 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:19.017666 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.017387 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:24:19.017666 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.017451 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:21.017437098 +0000 UTC m=+43.732656049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : secret "insights-runtime-extractor-tls" not found Jul 06 16:24:19.017762 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.017745 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3238e000-945c-4464-aa32-2f2e0843d959-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:19.319764 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.319741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:19.319950 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.319931 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:24:19.319996 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.319947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:19.320039 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.319998 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls podName:a877b735-824f-46f9-b2e4-016dc8bd2725 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:27.319980139 +0000 UTC m=+50.035199094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls") pod "dns-default-qtdqx" (UID: "a877b735-824f-46f9-b2e4-016dc8bd2725") : secret "dns-default-metrics-tls" not found Jul 06 16:24:19.320089 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.320038 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:24:19.320089 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.320050 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc8cdfbd7-8k66g: secret "image-registry-tls" not found Jul 06 16:24:19.320160 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.320091 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls podName:8269cb04-9e02-4190-b15b-f64a1fc59bbc nodeName:}" failed. No retries permitted until 2026-07-06 16:24:27.320080747 +0000 UTC m=+50.035299698 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls") pod "image-registry-fc8cdfbd7-8k66g" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc") : secret "image-registry-tls" not found Jul 06 16:24:19.421062 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.421020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:19.421062 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.421077 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:19.421296 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.421105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:19.421296 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.421206 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 16:24:19.421296 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.421223 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:27.421204784 +0000 UTC m=+50.136423737 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : configmap references non-existent config key: service-ca.crt Jul 06 16:24:19.421296 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.421258 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:27.421246771 +0000 UTC m=+50.136465724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : secret "router-metrics-certs-default" not found Jul 06 16:24:19.421296 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.421215 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:19.421296 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:19.421286 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls podName:ba335293-eebb-4b84-87aa-84258e6c4111 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:27.421280655 +0000 UTC m=+50.136499606 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jk7mk" (UID: "ba335293-eebb-4b84-87aa-84258e6c4111") : secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:19.536002 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.535972 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-796kr_bede19a7-4918-4b48-8bf0-f5ade4d53795/node-ca/0.log" Jul 06 16:24:19.955081 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.955047 2563 generic.go:356] "Generic (PLEG): container finished" podID="03734ae6-5b46-440f-99a2-6eac216a3675" containerID="8be883e5b3755b1a82c7bfe328dc950290b1a5afafa63118ab61745d2a77513d" exitCode=0 Jul 06 16:24:19.955236 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:19.955110 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerDied","Data":"8be883e5b3755b1a82c7bfe328dc950290b1a5afafa63118ab61745d2a77513d"} Jul 06 16:24:20.138353 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:20.138321 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5p5zj_73c3b173-35b2-41ac-bd6f-e568f1229742/serve-healthcheck-canary/0.log" Jul 06 16:24:20.962606 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:20.962571 2563 generic.go:356] "Generic (PLEG): container finished" podID="03734ae6-5b46-440f-99a2-6eac216a3675" containerID="aa59627a7540b49cdc360981bca7ec3ed17d17366d339f187914caddb7920664" exitCode=0 Jul 06 16:24:20.962764 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:20.962628 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerDied","Data":"aa59627a7540b49cdc360981bca7ec3ed17d17366d339f187914caddb7920664"} Jul 06 16:24:21.038911 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:21.038859 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:21.039044 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:21.039025 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:24:21.039100 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:21.039083 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:25.039063411 +0000 UTC m=+47.754282363 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : secret "insights-runtime-extractor-tls" not found Jul 06 16:24:21.967701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:21.967665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-57tjt" event={"ID":"03734ae6-5b46-440f-99a2-6eac216a3675","Type":"ContainerStarted","Data":"40eadb2141f47fca98cd4c814b9d4c8a35abdcbd296266ab348e8037dbac1894"} Jul 06 16:24:21.993882 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:21.993831 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-57tjt" podStartSLOduration=8.681525199 podStartE2EDuration="43.993816229s" totalImagesPullingTime="35.31229103s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:21.993786997 +0000 UTC m=+44.709005982" watchObservedRunningTime="2026-07-06 16:24:21.993816229 +0000 UTC m=+44.709035219" Jul 06 16:24:23.097997 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.097959 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69"] Jul 06 16:24:23.140006 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.139967 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69"] Jul 06 16:24:23.140149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.140087 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" Jul 06 16:24:23.145008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.144984 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-txv4b\"" Jul 06 16:24:23.145342 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.144985 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 06 16:24:23.145680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.145655 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 06 16:24:23.153699 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.153676 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q78jr\" (UniqueName: \"kubernetes.io/projected/e63fb0d8-5cb8-4668-8667-0ea1929f777a-kube-api-access-q78jr\") pod \"migrator-d57d58cc9-5th69\" (UID: \"e63fb0d8-5cb8-4668-8667-0ea1929f777a\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" Jul 06 16:24:23.255015 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.254976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q78jr\" (UniqueName: \"kubernetes.io/projected/e63fb0d8-5cb8-4668-8667-0ea1929f777a-kube-api-access-q78jr\") pod \"migrator-d57d58cc9-5th69\" (UID: \"e63fb0d8-5cb8-4668-8667-0ea1929f777a\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" Jul 06 16:24:23.268475 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.268452 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q78jr\" (UniqueName: \"kubernetes.io/projected/e63fb0d8-5cb8-4668-8667-0ea1929f777a-kube-api-access-q78jr\") pod \"migrator-d57d58cc9-5th69\" (UID: \"e63fb0d8-5cb8-4668-8667-0ea1929f777a\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" Jul 06 16:24:23.452388 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.452296 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" Jul 06 16:24:23.594627 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.594583 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69"] Jul 06 16:24:23.608419 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:23.608389 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode63fb0d8_5cb8_4668_8667_0ea1929f777a.slice/crio-9d8c6bee30fd91b67b9391d9bc274d2473e0837373976ab2e61ac2c4928bc57d WatchSource:0}: Error finding container 9d8c6bee30fd91b67b9391d9bc274d2473e0837373976ab2e61ac2c4928bc57d: Status 404 returned error can't find the container with id 9d8c6bee30fd91b67b9391d9bc274d2473e0837373976ab2e61ac2c4928bc57d Jul 06 16:24:23.973727 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:23.973688 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" event={"ID":"e63fb0d8-5cb8-4668-8667-0ea1929f777a","Type":"ContainerStarted","Data":"9d8c6bee30fd91b67b9391d9bc274d2473e0837373976ab2e61ac2c4928bc57d"} Jul 06 16:24:25.072742 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:25.072697 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:25.073182 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:25.072897 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:24:25.073182 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:25.072977 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:33.072955736 +0000 UTC m=+55.788174688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : secret "insights-runtime-extractor-tls" not found Jul 06 16:24:25.980105 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:25.980028 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" event={"ID":"e63fb0d8-5cb8-4668-8667-0ea1929f777a","Type":"ContainerStarted","Data":"1f18db123258f8b221df7ffbc623e0220edfb9cbaa9883ee26f836037e61fa1a"} Jul 06 16:24:25.980105 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:25.980077 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" event={"ID":"e63fb0d8-5cb8-4668-8667-0ea1929f777a","Type":"ContainerStarted","Data":"1b5be6f0bd0072c9cdc3c03abe856ca9419d7484cc323df6758dc5bf677cc993"} Jul 06 16:24:26.002522 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:26.002470 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-5th69" podStartSLOduration=0.935692326 podStartE2EDuration="3.002453745s" totalImagesPullingTime="2.066761419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:26.000904694 +0000 UTC m=+48.716123692" watchObservedRunningTime="2026-07-06 16:24:26.002453745 +0000 UTC m=+48.717672718" Jul 06 16:24:27.393481 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:27.393440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:27.393893 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:27.393514 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:27.393893 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.393606 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 06 16:24:27.393893 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.393634 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 06 16:24:27.393893 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.393645 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-fc8cdfbd7-8k66g: secret "image-registry-tls" not found Jul 06 16:24:27.393893 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.393671 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls podName:a877b735-824f-46f9-b2e4-016dc8bd2725 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.393655619 +0000 UTC m=+66.108874571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls") pod "dns-default-qtdqx" (UID: "a877b735-824f-46f9-b2e4-016dc8bd2725") : secret "dns-default-metrics-tls" not found Jul 06 16:24:27.393893 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.393686 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls podName:8269cb04-9e02-4190-b15b-f64a1fc59bbc nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.393679901 +0000 UTC m=+66.108898852 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls") pod "image-registry-fc8cdfbd7-8k66g" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc") : secret "image-registry-tls" not found Jul 06 16:24:27.494578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:27.494541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:27.494578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:27.494584 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:27.494859 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:27.494606 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:27.494859 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.494717 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 06 16:24:27.494859 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.494813 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.494769459 +0000 UTC m=+66.209988425 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : secret "router-metrics-certs-default" not found Jul 06 16:24:27.494859 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.494726 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:27.494859 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.494835 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle podName:d1148015-927a-46cb-8fa4-e4520fc74763 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.494824489 +0000 UTC m=+66.210043444 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle") pod "router-default-996dbbc49-g98f5" (UID: "d1148015-927a-46cb-8fa4-e4520fc74763") : configmap references non-existent config key: service-ca.crt Jul 06 16:24:27.495032 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:27.494867 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls podName:ba335293-eebb-4b84-87aa-84258e6c4111 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:43.494852101 +0000 UTC m=+66.210071054 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-jk7mk" (UID: "ba335293-eebb-4b84-87aa-84258e6c4111") : secret "cluster-monitoring-operator-tls" not found Jul 06 16:24:33.139660 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:33.139612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:33.140210 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:33.139757 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 06 16:24:33.140210 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:33.139841 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls podName:3238e000-945c-4464-aa32-2f2e0843d959 nodeName:}" failed. No retries permitted until 2026-07-06 16:24:49.139826118 +0000 UTC m=+71.855045070 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sdkwl" (UID: "3238e000-945c-4464-aa32-2f2e0843d959") : secret "insights-runtime-extractor-tls" not found Jul 06 16:24:37.913477 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:37.913448 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56nhz" Jul 06 16:24:43.434022 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.433987 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:43.434476 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.434058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:43.436415 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.436392 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a877b735-824f-46f9-b2e4-016dc8bd2725-metrics-tls\") pod \"dns-default-qtdqx\" (UID: \"a877b735-824f-46f9-b2e4-016dc8bd2725\") " pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:43.436471 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.436442 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"image-registry-fc8cdfbd7-8k66g\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:43.535200 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.535161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:43.535200 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.535207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:43.535422 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.535234 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:43.535936 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.535908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d1148015-927a-46cb-8fa4-e4520fc74763-service-ca-bundle\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:43.537526 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.537501 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d1148015-927a-46cb-8fa4-e4520fc74763-metrics-certs\") pod \"router-default-996dbbc49-g98f5\" (UID: \"d1148015-927a-46cb-8fa4-e4520fc74763\") " pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:43.537660 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.537638 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/ba335293-eebb-4b84-87aa-84258e6c4111-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-jk7mk\" (UID: \"ba335293-eebb-4b84-87aa-84258e6c4111\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:43.664163 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.664130 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qh9j6\"" Jul 06 16:24:43.672025 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.672001 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:43.679732 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.679715 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wmlq9\"" Jul 06 16:24:43.687495 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.687440 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:43.699381 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.699354 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-9stxg\"" Jul 06 16:24:43.706955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.706937 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" Jul 06 16:24:43.721899 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.721258 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-68jqv\"" Jul 06 16:24:43.728978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.728727 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:43.737442 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.736749 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:43.737442 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.736874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:43.742250 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.741712 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 06 16:24:43.742250 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.742048 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 06 16:24:43.750526 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.750467 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/65242094-074d-41d5-a8e9-d22a5c636b9f-original-pull-secret\") pod \"global-pull-secret-syncer-m285v\" (UID: \"65242094-074d-41d5-a8e9-d22a5c636b9f\") " pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:43.751052 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.751020 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f864ebe-e103-4a19-96c9-cd9d2beba99f-metrics-certs\") pod \"network-metrics-daemon-d6nms\" (UID: \"0f864ebe-e103-4a19-96c9-cd9d2beba99f\") " pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:43.833732 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.833700 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-fc8cdfbd7-8k66g"] Jul 06 16:24:43.838306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.838278 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:43.840901 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:43.840876 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8269cb04_9e02_4190_b15b_f64a1fc59bbc.slice/crio-059588c4f4e75509a406ca59f0e8c9fe553eb5122412548f1a87f65ed2fc9a47 WatchSource:0}: Error finding container 059588c4f4e75509a406ca59f0e8c9fe553eb5122412548f1a87f65ed2fc9a47: Status 404 returned error can't find the container with id 059588c4f4e75509a406ca59f0e8c9fe553eb5122412548f1a87f65ed2fc9a47 Jul 06 16:24:43.844103 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.844084 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 06 16:24:43.851679 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.851653 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 06 16:24:43.861482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.861460 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qtdqx"] Jul 06 16:24:43.861848 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.861820 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5zn\" (UniqueName: \"kubernetes.io/projected/6b8a8d6b-f434-4396-b90e-aae39e897dab-kube-api-access-5v5zn\") pod \"network-check-target-xxtrg\" (UID: \"6b8a8d6b-f434-4396-b90e-aae39e897dab\") " pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:43.869779 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:43.869756 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda877b735_824f_46f9_b2e4_016dc8bd2725.slice/crio-13b2d69e3ed374744f0fddbfbe60aea91e4f5bc1226c30fbee6f88d62fd46d6a WatchSource:0}: Error finding container 13b2d69e3ed374744f0fddbfbe60aea91e4f5bc1226c30fbee6f88d62fd46d6a: Status 404 returned error can't find the container with id 13b2d69e3ed374744f0fddbfbe60aea91e4f5bc1226c30fbee6f88d62fd46d6a Jul 06 16:24:43.876371 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.876351 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk"] Jul 06 16:24:43.901077 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.901059 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-6db6n\"" Jul 06 16:24:43.903591 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.903567 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-m285v" Jul 06 16:24:43.907647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.907592 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-996dbbc49-g98f5"] Jul 06 16:24:43.908214 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.908086 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:43.911481 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.911461 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hn2l7\"" Jul 06 16:24:43.912675 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:43.912652 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1148015_927a_46cb_8fa4_e4520fc74763.slice/crio-d7a8df627d59427aaf426c038d7ea451ad41a455ebe5ca3c924946ef181e840c WatchSource:0}: Error finding container d7a8df627d59427aaf426c038d7ea451ad41a455ebe5ca3c924946ef181e840c: Status 404 returned error can't find the container with id d7a8df627d59427aaf426c038d7ea451ad41a455ebe5ca3c924946ef181e840c Jul 06 16:24:43.918990 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:43.918968 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-d6nms" Jul 06 16:24:44.028773 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.028742 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-996dbbc49-g98f5" event={"ID":"d1148015-927a-46cb-8fa4-e4520fc74763","Type":"ContainerStarted","Data":"0df01a86aa61237e20223e9f19f0d262af7c68d974d1ea23f29e2983983a5060"} Jul 06 16:24:44.029014 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.028831 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-996dbbc49-g98f5" event={"ID":"d1148015-927a-46cb-8fa4-e4520fc74763","Type":"ContainerStarted","Data":"d7a8df627d59427aaf426c038d7ea451ad41a455ebe5ca3c924946ef181e840c"} Jul 06 16:24:44.030168 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.030098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" event={"ID":"ba335293-eebb-4b84-87aa-84258e6c4111","Type":"ContainerStarted","Data":"754b658ed4aabc2adbe508f01184c9e8555cf74e668ec1de03f4100ceda40061"} Jul 06 16:24:44.031537 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.031467 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtdqx" event={"ID":"a877b735-824f-46f9-b2e4-016dc8bd2725","Type":"ContainerStarted","Data":"13b2d69e3ed374744f0fddbfbe60aea91e4f5bc1226c30fbee6f88d62fd46d6a"} Jul 06 16:24:44.034207 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.033612 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" event={"ID":"8269cb04-9e02-4190-b15b-f64a1fc59bbc","Type":"ContainerStarted","Data":"0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc"} Jul 06 16:24:44.034207 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.033651 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" event={"ID":"8269cb04-9e02-4190-b15b-f64a1fc59bbc","Type":"ContainerStarted","Data":"059588c4f4e75509a406ca59f0e8c9fe553eb5122412548f1a87f65ed2fc9a47"} Jul 06 16:24:44.034207 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.034159 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:24:44.059491 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.059045 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-996dbbc49-g98f5" podStartSLOduration=37.059022206 podStartE2EDuration="37.059022206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:44.057337626 +0000 UTC m=+66.772556630" watchObservedRunningTime="2026-07-06 16:24:44.059022206 +0000 UTC m=+66.774241181" Jul 06 16:24:44.082609 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.082516 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-m285v"] Jul 06 16:24:44.085104 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:44.085064 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65242094_074d_41d5_a8e9_d22a5c636b9f.slice/crio-eee01e4e7905510c23c8f93aa5b574e46980d938332031ee08a5bd855f4df9d9 WatchSource:0}: Error finding container eee01e4e7905510c23c8f93aa5b574e46980d938332031ee08a5bd855f4df9d9: Status 404 returned error can't find the container with id eee01e4e7905510c23c8f93aa5b574e46980d938332031ee08a5bd855f4df9d9 Jul 06 16:24:44.092124 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.092074 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" podStartSLOduration=67.092053497 podStartE2EDuration="1m7.092053497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:44.091422897 +0000 UTC m=+66.806641892" watchObservedRunningTime="2026-07-06 16:24:44.092053497 +0000 UTC m=+66.807272475" Jul 06 16:24:44.103031 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.103009 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-d6nms"] Jul 06 16:24:44.293647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.293577 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xxtrg"] Jul 06 16:24:44.297577 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:44.297552 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8a8d6b_f434_4396_b90e_aae39e897dab.slice/crio-20dbeaabff4bab851f3e16f8754c72e3191ed6616baa4c779c8077e58885d37d WatchSource:0}: Error finding container 20dbeaabff4bab851f3e16f8754c72e3191ed6616baa4c779c8077e58885d37d: Status 404 returned error can't find the container with id 20dbeaabff4bab851f3e16f8754c72e3191ed6616baa4c779c8077e58885d37d Jul 06 16:24:44.730006 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.729929 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:44.733273 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:44.733247 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:45.040845 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:45.040643 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6nms" event={"ID":"0f864ebe-e103-4a19-96c9-cd9d2beba99f","Type":"ContainerStarted","Data":"3b056d48d723a5b04f2e524225e926b84ff0ec0a73590677fd536b97fb0ca4eb"} Jul 06 16:24:45.042926 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:45.042773 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxtrg" event={"ID":"6b8a8d6b-f434-4396-b90e-aae39e897dab","Type":"ContainerStarted","Data":"20dbeaabff4bab851f3e16f8754c72e3191ed6616baa4c779c8077e58885d37d"} Jul 06 16:24:45.046300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:45.046066 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m285v" event={"ID":"65242094-074d-41d5-a8e9-d22a5c636b9f","Type":"ContainerStarted","Data":"eee01e4e7905510c23c8f93aa5b574e46980d938332031ee08a5bd855f4df9d9"} Jul 06 16:24:45.046916 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:45.046491 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:45.049050 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:45.048835 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-996dbbc49-g98f5" Jul 06 16:24:46.051575 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.051347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6nms" event={"ID":"0f864ebe-e103-4a19-96c9-cd9d2beba99f","Type":"ContainerStarted","Data":"4ef986811143cafa952e54ad9ac2266a8a6b2288c3707bbedff15b8918a6ca84"} Jul 06 16:24:46.051575 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.051412 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-d6nms" event={"ID":"0f864ebe-e103-4a19-96c9-cd9d2beba99f","Type":"ContainerStarted","Data":"d94e88de2edbbca8eb2f0298c8d9717a3ca85a513743a70f33bd1278e3f9e23e"} Jul 06 16:24:46.109957 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.108867 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-d6nms" podStartSLOduration=66.920716274 podStartE2EDuration="1m8.108842447s" totalImagesPullingTime="1.188126173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:46.107937702 +0000 UTC m=+68.823156704" watchObservedRunningTime="2026-07-06 16:24:46.108842447 +0000 UTC m=+68.824061421" Jul 06 16:24:46.112473 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.111508 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt"] Jul 06 16:24:46.117331 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.117198 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.123149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.122313 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lxgcj\"" Jul 06 16:24:46.123149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.122661 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 06 16:24:46.130885 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.128967 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 06 16:24:46.130885 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.130831 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt"] Jul 06 16:24:46.147087 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.147046 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz"] Jul 06 16:24:46.151613 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.150575 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.155190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.154943 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 06 16:24:46.155190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.154945 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 06 16:24:46.155190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.155097 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 06 16:24:46.156827 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.156092 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 06 16:24:46.156827 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.156360 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 06 16:24:46.156827 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.156547 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 06 16:24:46.156827 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.156679 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 06 16:24:46.181486 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.181433 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz"] Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.260939 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9fg2\" (UniqueName: \"kubernetes.io/projected/a33172d3-23e2-469c-b8e9-8ff34035c549-kube-api-access-d9fg2\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261046 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e51b6357-87fc-4d48-a162-054f00096e6b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xpvrt\" (UID: \"e51b6357-87fc-4d48-a162-054f00096e6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261071 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e51b6357-87fc-4d48-a162-054f00096e6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xpvrt\" (UID: \"e51b6357-87fc-4d48-a162-054f00096e6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261100 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-ca\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261163 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33172d3-23e2-469c-b8e9-8ff34035c549-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.261347 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.261236 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-hub\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362044 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e51b6357-87fc-4d48-a162-054f00096e6b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xpvrt\" (UID: \"e51b6357-87fc-4d48-a162-054f00096e6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e51b6357-87fc-4d48-a162-054f00096e6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xpvrt\" (UID: \"e51b6357-87fc-4d48-a162-054f00096e6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362160 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-ca\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33172d3-23e2-469c-b8e9-8ff34035c549-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362295 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-hub\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.362856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.362339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9fg2\" (UniqueName: \"kubernetes.io/projected/a33172d3-23e2-469c-b8e9-8ff34035c549-kube-api-access-d9fg2\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.363908 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.363567 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e51b6357-87fc-4d48-a162-054f00096e6b-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-xpvrt\" (UID: \"e51b6357-87fc-4d48-a162-054f00096e6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.363908 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.363683 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/a33172d3-23e2-469c-b8e9-8ff34035c549-additional-service-ca\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.366341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.366314 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.366694 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.366645 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-hub\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.367220 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.367175 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-ca\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.367354 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.367332 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e51b6357-87fc-4d48-a162-054f00096e6b-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-xpvrt\" (UID: \"e51b6357-87fc-4d48-a162-054f00096e6b\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.369888 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.369863 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/a33172d3-23e2-469c-b8e9-8ff34035c549-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.373575 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.373549 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9fg2\" (UniqueName: \"kubernetes.io/projected/a33172d3-23e2-469c-b8e9-8ff34035c549-kube-api-access-d9fg2\") pod \"cluster-proxy-proxy-agent-9bdb6c957-4w8lz\" (UID: \"a33172d3-23e2-469c-b8e9-8ff34035c549\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:46.434504 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.434470 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" Jul 06 16:24:46.473880 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:46.473573 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" Jul 06 16:24:49.188498 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:49.188458 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:49.191382 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:49.191346 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3238e000-945c-4464-aa32-2f2e0843d959-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sdkwl\" (UID: \"3238e000-945c-4464-aa32-2f2e0843d959\") " pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:49.341594 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:49.341559 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gb7wf\"" Jul 06 16:24:49.349254 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:49.349225 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sdkwl" Jul 06 16:24:50.513357 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.513264 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sdkwl"] Jul 06 16:24:50.551222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.550961 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt"] Jul 06 16:24:50.555812 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.552707 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz"] Jul 06 16:24:50.560202 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:50.560170 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51b6357_87fc_4d48_a162_054f00096e6b.slice/crio-9aa96fab7ee51a1f7f4c15dd59a8a2bf0a515593c14f2019111ef7874c3a6ebf WatchSource:0}: Error finding container 9aa96fab7ee51a1f7f4c15dd59a8a2bf0a515593c14f2019111ef7874c3a6ebf: Status 404 returned error can't find the container with id 9aa96fab7ee51a1f7f4c15dd59a8a2bf0a515593c14f2019111ef7874c3a6ebf Jul 06 16:24:50.561148 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:50.561128 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda33172d3_23e2_469c_b8e9_8ff34035c549.slice/crio-cde2c65ade5480c61fe24a6ae6a7fff4e3c371f647743b13550de2a92f452182 WatchSource:0}: Error finding container cde2c65ade5480c61fe24a6ae6a7fff4e3c371f647743b13550de2a92f452182: Status 404 returned error can't find the container with id cde2c65ade5480c61fe24a6ae6a7fff4e3c371f647743b13550de2a92f452182 Jul 06 16:24:50.987782 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.987746 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs"] Jul 06 16:24:50.990081 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.990053 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:50.993260 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.993233 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-wthzw\"" Jul 06 16:24:50.993470 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:50.993383 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 06 16:24:51.001582 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.001558 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs"] Jul 06 16:24:51.073060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.073020 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtdqx" event={"ID":"a877b735-824f-46f9-b2e4-016dc8bd2725","Type":"ContainerStarted","Data":"48544402c9ba13cdc773a8d8b6469dff6cd5513cb3c3fa9436694ec796da2784"} Jul 06 16:24:51.073244 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.073091 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qtdqx" event={"ID":"a877b735-824f-46f9-b2e4-016dc8bd2725","Type":"ContainerStarted","Data":"fad0bddd5f2f1badbb65965df694eae12ae03364b994bc91ce98ca1e6ea0fca0"} Jul 06 16:24:51.073443 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.073378 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qtdqx" Jul 06 16:24:51.075578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.075508 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xxtrg" event={"ID":"6b8a8d6b-f434-4396-b90e-aae39e897dab","Type":"ContainerStarted","Data":"ec90c719fc37ce2349a91c06b9311756e9b5185f9401d85fafd94a9d859d9f02"} Jul 06 16:24:51.075982 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.075931 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:24:51.077630 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.077570 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" event={"ID":"a33172d3-23e2-469c-b8e9-8ff34035c549","Type":"ContainerStarted","Data":"cde2c65ade5480c61fe24a6ae6a7fff4e3c371f647743b13550de2a92f452182"} Jul 06 16:24:51.081449 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.081060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" event={"ID":"ba335293-eebb-4b84-87aa-84258e6c4111","Type":"ContainerStarted","Data":"3e4c2990e1222d475a6175a4be59168748723a312e6174a15643016fe22321dc"} Jul 06 16:24:51.083545 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.083499 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-m285v" event={"ID":"65242094-074d-41d5-a8e9-d22a5c636b9f","Type":"ContainerStarted","Data":"afc473bc56cbf5da7edefcca9fc06ccaa5bc591524d5f2c57debb4fd33175aea"} Jul 06 16:24:51.084868 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.084805 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdkwl" event={"ID":"3238e000-945c-4464-aa32-2f2e0843d959","Type":"ContainerStarted","Data":"147588576f4c1b71e5dfbcf45757c5950df9452a67d1985c0ddc37bc60da02e0"} Jul 06 16:24:51.087395 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.087348 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" event={"ID":"e51b6357-87fc-4d48-a162-054f00096e6b","Type":"ContainerStarted","Data":"9aa96fab7ee51a1f7f4c15dd59a8a2bf0a515593c14f2019111ef7874c3a6ebf"} Jul 06 16:24:51.095840 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.095651 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qtdqx" podStartSLOduration=33.634369476 podStartE2EDuration="40.095630923s" totalImagesPullingTime="6.461261447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:51.094101442 +0000 UTC m=+73.809320417" watchObservedRunningTime="2026-07-06 16:24:51.095630923 +0000 UTC m=+73.810849878" Jul 06 16:24:51.107401 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.107087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/31373c34-0755-4552-8ed3-0570d9fd37db-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kjvrs\" (UID: \"31373c34-0755-4552-8ed3-0570d9fd37db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:51.115963 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.115043 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-jk7mk" podStartSLOduration=37.679939734 podStartE2EDuration="44.115025059s" totalImagesPullingTime="6.435085325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:51.113757112 +0000 UTC m=+73.828976086" watchObservedRunningTime="2026-07-06 16:24:51.115025059 +0000 UTC m=+73.830244034" Jul 06 16:24:51.132768 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.132704 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xxtrg" podStartSLOduration=67.096832797 podStartE2EDuration="1m13.132682777s" totalImagesPullingTime="6.03584998s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:51.13228285 +0000 UTC m=+73.847501909" watchObservedRunningTime="2026-07-06 16:24:51.132682777 +0000 UTC m=+73.847901751" Jul 06 16:24:51.209405 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.207966 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/31373c34-0755-4552-8ed3-0570d9fd37db-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kjvrs\" (UID: \"31373c34-0755-4552-8ed3-0570d9fd37db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:51.209405 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:51.209061 2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 06 16:24:51.209405 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:24:51.209127 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31373c34-0755-4552-8ed3-0570d9fd37db-tls-certificates podName:31373c34-0755-4552-8ed3-0570d9fd37db nodeName:}" failed. No retries permitted until 2026-07-06 16:24:51.709108733 +0000 UTC m=+74.424327695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/31373c34-0755-4552-8ed3-0570d9fd37db-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-kjvrs" (UID: "31373c34-0755-4552-8ed3-0570d9fd37db") : secret "prometheus-operator-admission-webhook-tls" not found Jul 06 16:24:51.714835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.714777 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/31373c34-0755-4552-8ed3-0570d9fd37db-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kjvrs\" (UID: \"31373c34-0755-4552-8ed3-0570d9fd37db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:51.720422 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.720359 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/31373c34-0755-4552-8ed3-0570d9fd37db-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-kjvrs\" (UID: \"31373c34-0755-4552-8ed3-0570d9fd37db\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:51.902158 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:51.902123 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:52.403957 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:52.403879 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-m285v" podStartSLOduration=68.147775577 podStartE2EDuration="1m14.403855049s" totalImagesPullingTime="6.256079472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:23:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:51.152154519 +0000 UTC m=+73.867373514" watchObservedRunningTime="2026-07-06 16:24:52.403855049 +0000 UTC m=+75.119074040" Jul 06 16:24:52.404205 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:52.404181 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs"] Jul 06 16:24:52.414699 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:24:52.409113 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31373c34_0755_4552_8ed3_0570d9fd37db.slice/crio-fd098fecc51959d3c7c06f7d543ecb993227ec722fa4d7be83d3b5c329abc794 WatchSource:0}: Error finding container fd098fecc51959d3c7c06f7d543ecb993227ec722fa4d7be83d3b5c329abc794: Status 404 returned error can't find the container with id fd098fecc51959d3c7c06f7d543ecb993227ec722fa4d7be83d3b5c329abc794 Jul 06 16:24:53.099141 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:53.099100 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" event={"ID":"e51b6357-87fc-4d48-a162-054f00096e6b","Type":"ContainerStarted","Data":"e8fa9646dd0bce859536a2d9164dbf8fdeb3e2e04597ff9978cbc28d9bad772e"} Jul 06 16:24:53.100737 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:53.100677 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" event={"ID":"31373c34-0755-4552-8ed3-0570d9fd37db","Type":"ContainerStarted","Data":"fd098fecc51959d3c7c06f7d543ecb993227ec722fa4d7be83d3b5c329abc794"} Jul 06 16:24:53.125676 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:53.125621 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-xpvrt" podStartSLOduration=5.410395422 podStartE2EDuration="7.125600635s" totalImagesPullingTime="1.715205213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:53.123000911 +0000 UTC m=+75.838219913" watchObservedRunningTime="2026-07-06 16:24:53.125600635 +0000 UTC m=+75.840819612" Jul 06 16:24:57.119723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.119690 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdkwl" event={"ID":"3238e000-945c-4464-aa32-2f2e0843d959","Type":"ContainerStarted","Data":"46140d784fb0fd836fd45f93279f04ddb788db5c32b7179d44928a3e00e23cae"} Jul 06 16:24:57.121197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.121174 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" event={"ID":"31373c34-0755-4552-8ed3-0570d9fd37db","Type":"ContainerStarted","Data":"24ba48ff09b01cabdb6437c202b6e6b9324d61cda2210a6921d7a27838ef3980"} Jul 06 16:24:57.121362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.121335 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:57.123311 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.123289 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" event={"ID":"a33172d3-23e2-469c-b8e9-8ff34035c549","Type":"ContainerStarted","Data":"a84f58a3d3a0811af31fe8ee42578ec458c2d2d364cd5912125374da34e0d3be"} Jul 06 16:24:57.123412 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.123324 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" event={"ID":"a33172d3-23e2-469c-b8e9-8ff34035c549","Type":"ContainerStarted","Data":"a7c8d50ed3dbfb16c0101c92232bb4c9c8caa6dc9205ea762a5ba4fcc4a8b960"} Jul 06 16:24:57.123412 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.123342 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" event={"ID":"a33172d3-23e2-469c-b8e9-8ff34035c549","Type":"ContainerStarted","Data":"635df2e50c1f40f26c58050960058275e7e0e99ecd27c45064a898af28e47b27"} Jul 06 16:24:57.126193 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.126174 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" Jul 06 16:24:57.158180 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.158097 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-kjvrs" podStartSLOduration=2.681910385 podStartE2EDuration="7.158083251s" totalImagesPullingTime="4.476172866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:57.158053161 +0000 UTC m=+79.873272179" watchObservedRunningTime="2026-07-06 16:24:57.158083251 +0000 UTC m=+79.873302215" Jul 06 16:24:57.242248 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:57.242196 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-9bdb6c957-4w8lz" podStartSLOduration=4.907770704 podStartE2EDuration="11.242178874s" totalImagesPullingTime="6.33440817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:24:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:24:57.241979025 +0000 UTC m=+79.957197999" watchObservedRunningTime="2026-07-06 16:24:57.242178874 +0000 UTC m=+79.957397852" Jul 06 16:24:58.129613 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:24:58.129573 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdkwl" event={"ID":"3238e000-945c-4464-aa32-2f2e0843d959","Type":"ContainerStarted","Data":"c57f10ac053ae981df17fc84c268d1ccf19caa412fe33ff362efae13b05a84f5"} Jul 06 16:25:01.096251 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:01.096222 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qtdqx" Jul 06 16:25:01.145345 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:01.145314 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sdkwl" event={"ID":"3238e000-945c-4464-aa32-2f2e0843d959","Type":"ContainerStarted","Data":"ff9f8c5f6687665ccf0a542a2ce9e04fae556f57edc0cdfe399341d665143cab"} Jul 06 16:25:01.176033 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:01.175980 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sdkwl" podStartSLOduration=34.769973254 podStartE2EDuration="44.175962207s" totalImagesPullingTime="9.405988953s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 16:24:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:25:01.175431835 +0000 UTC m=+83.890650811" watchObservedRunningTime="2026-07-06 16:25:01.175962207 +0000 UTC m=+83.891181180" Jul 06 16:25:03.676983 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:03.676936 2563 patch_prober.go:28] interesting pod/image-registry-fc8cdfbd7-8k66g container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 06 16:25:03.677435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:03.677020 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" podUID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:25:06.056140 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.056109 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:25:06.124425 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.124390 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ld8fk"] Jul 06 16:25:06.128720 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.128698 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.146820 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:25:06.146770 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"node-exporter-tls\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-monitoring\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" Jul 06 16:25:06.161425 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.161401 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 06 16:25:06.161681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.161666 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hnwqs\"" Jul 06 16:25:06.171870 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.171848 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 06 16:25:06.220184 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.220157 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 06 16:25:06.227927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.227900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-root\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.227984 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-wtmp\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228072 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a0d007-fab8-41f5-abe1-25ea08c88780-metrics-client-ca\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228269 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228139 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-tls\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228269 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-textfile\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228329 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxg9\" (UniqueName: \"kubernetes.io/projected/e4a0d007-fab8-41f5-abe1-25ea08c88780-kube-api-access-swxg9\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228405 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228387 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-sys\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.228471 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.228445 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-accelerators-collector-config\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.329919 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.329818 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-wtmp\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.329919 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.329880 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.329919 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.329904 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a0d007-fab8-41f5-abe1-25ea08c88780-metrics-client-ca\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.329936 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-tls\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.329953 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-wtmp\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330054 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-textfile\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxg9\" (UniqueName: \"kubernetes.io/projected/e4a0d007-fab8-41f5-abe1-25ea08c88780-kube-api-access-swxg9\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330127 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-sys\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330171 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-accelerators-collector-config\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330454 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330217 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-root\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330454 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330233 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-sys\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330454 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330325 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e4a0d007-fab8-41f5-abe1-25ea08c88780-root\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330454 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330391 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-textfile\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330574 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330512 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4a0d007-fab8-41f5-abe1-25ea08c88780-metrics-client-ca\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.330729 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.330709 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-accelerators-collector-config\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.332573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.332551 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:06.350089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:06.350063 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxg9\" (UniqueName: \"kubernetes.io/projected/e4a0d007-fab8-41f5-abe1-25ea08c88780-kube-api-access-swxg9\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:07.330729 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:25:07.330687 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: failed to sync secret cache: timed out waiting for the condition Jul 06 16:25:07.331153 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:25:07.330783 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-tls podName:e4a0d007-fab8-41f5-abe1-25ea08c88780 nodeName:}" failed. No retries permitted until 2026-07-06 16:25:07.830763195 +0000 UTC m=+90.545982151 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-tls") pod "node-exporter-ld8fk" (UID: "e4a0d007-fab8-41f5-abe1-25ea08c88780") : failed to sync secret cache: timed out waiting for the condition Jul 06 16:25:07.533938 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:07.533907 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 06 16:25:07.844612 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:07.844575 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-tls\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:07.846935 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:07.846912 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e4a0d007-fab8-41f5-abe1-25ea08c88780-node-exporter-tls\") pod \"node-exporter-ld8fk\" (UID: \"e4a0d007-fab8-41f5-abe1-25ea08c88780\") " pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:07.937073 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:07.937039 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ld8fk" Jul 06 16:25:07.948861 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:25:07.948827 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a0d007_fab8_41f5_abe1_25ea08c88780.slice/crio-8b867bc89ef3319fe33dfa4384109081722256dd350e062d5ad53fbda02a8265 WatchSource:0}: Error finding container 8b867bc89ef3319fe33dfa4384109081722256dd350e062d5ad53fbda02a8265: Status 404 returned error can't find the container with id 8b867bc89ef3319fe33dfa4384109081722256dd350e062d5ad53fbda02a8265 Jul 06 16:25:08.172072 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:08.171989 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld8fk" event={"ID":"e4a0d007-fab8-41f5-abe1-25ea08c88780","Type":"ContainerStarted","Data":"8b867bc89ef3319fe33dfa4384109081722256dd350e062d5ad53fbda02a8265"} Jul 06 16:25:09.177389 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:09.177353 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld8fk" event={"ID":"e4a0d007-fab8-41f5-abe1-25ea08c88780","Type":"ContainerStarted","Data":"8135fc1bddfb3a466af245b7d6030be0b001c58ef82845f06a486c1c1bc09891"} Jul 06 16:25:10.182969 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:10.182939 2563 generic.go:356] "Generic (PLEG): container finished" podID="e4a0d007-fab8-41f5-abe1-25ea08c88780" containerID="8135fc1bddfb3a466af245b7d6030be0b001c58ef82845f06a486c1c1bc09891" exitCode=0 Jul 06 16:25:10.183417 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:10.183024 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld8fk" event={"ID":"e4a0d007-fab8-41f5-abe1-25ea08c88780","Type":"ContainerDied","Data":"8135fc1bddfb3a466af245b7d6030be0b001c58ef82845f06a486c1c1bc09891"} Jul 06 16:25:11.192850 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:11.192806 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld8fk" event={"ID":"e4a0d007-fab8-41f5-abe1-25ea08c88780","Type":"ContainerStarted","Data":"260180fd8ddaee8e153b2e90a3abadd375417a450c6824fddfe09107e64276aa"} Jul 06 16:25:11.193256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:11.192889 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ld8fk" event={"ID":"e4a0d007-fab8-41f5-abe1-25ea08c88780","Type":"ContainerStarted","Data":"60cc71dc954f2785cd11ee23d7ad5500ddcb7ebf131d82295763ca0c04277dc6"} Jul 06 16:25:12.699857 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:12.699780 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ld8fk" podStartSLOduration=5.536721799 podStartE2EDuration="6.699761835s" totalImagesPullingTime="1.097153803s" totalInitContainerRuntime="65.886233ms" isStatefulPod=true podCreationTimestamp="2026-07-06 16:25:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:25:11.228856611 +0000 UTC m=+93.944075593" watchObservedRunningTime="2026-07-06 16:25:12.699761835 +0000 UTC m=+95.414980835" Jul 06 16:25:12.700736 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:12.700720 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fc8cdfbd7-8k66g"] Jul 06 16:25:23.102906 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:23.102873 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xxtrg" Jul 06 16:25:32.269138 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:32.269103 2563 generic.go:356] "Generic (PLEG): container finished" podID="d32c3f81-2740-490c-b120-f67350df8f95" containerID="b234d1b0f44603dc79269c064dccf7b81505e24e12166cf376c566a3b9b1a8e6" exitCode=0 Jul 06 16:25:32.269512 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:32.269180 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" event={"ID":"d32c3f81-2740-490c-b120-f67350df8f95","Type":"ContainerDied","Data":"b234d1b0f44603dc79269c064dccf7b81505e24e12166cf376c566a3b9b1a8e6"} Jul 06 16:25:32.269512 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:32.269508 2563 scope.go:117] "RemoveContainer" containerID="b234d1b0f44603dc79269c064dccf7b81505e24e12166cf376c566a3b9b1a8e6" Jul 06 16:25:33.274285 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:33.274257 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-nrkt5" event={"ID":"d32c3f81-2740-490c-b120-f67350df8f95","Type":"ContainerStarted","Data":"04931c5413e937b26989500f33c1d1660603da80dffdaf312a645033c20de8af"} Jul 06 16:25:37.720238 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:37.720198 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" podUID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" containerName="registry" containerID="cri-o://0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc" gracePeriod=30 Jul 06 16:25:37.953574 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:37.953551 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:25:38.038858 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.038761 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-image-registry-private-configuration\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.038858 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.038811 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-trusted-ca\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.038858 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.038856 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-certificates\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.039197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.038880 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-bound-sa-token\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.039197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.038908 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.039197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.039039 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8269cb04-9e02-4190-b15b-f64a1fc59bbc-ca-trust-extracted\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.039197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.039089 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2v8lh\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-kube-api-access-2v8lh\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.039197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.039180 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-installation-pull-secrets\") pod \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\" (UID: \"8269cb04-9e02-4190-b15b-f64a1fc59bbc\") " Jul 06 16:25:38.039456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.039350 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:25:38.039613 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.039585 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:25:38.041994 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.041950 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:25:38.042106 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.042007 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-kube-api-access-2v8lh" (OuterVolumeSpecName: "kube-api-access-2v8lh") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "kube-api-access-2v8lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:25:38.042106 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.042085 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:25:38.042197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.042109 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:25:38.042197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.042168 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:25:38.047810 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.047763 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8269cb04-9e02-4190-b15b-f64a1fc59bbc-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8269cb04-9e02-4190-b15b-f64a1fc59bbc" (UID: "8269cb04-9e02-4190-b15b-f64a1fc59bbc"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:25:38.140742 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140695 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-installation-pull-secrets\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.140742 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140739 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8269cb04-9e02-4190-b15b-f64a1fc59bbc-image-registry-private-configuration\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.140742 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140751 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-trusted-ca\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.140742 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140760 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-certificates\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.141035 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140769 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-bound-sa-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.141035 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140778 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-registry-tls\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.141035 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140785 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8269cb04-9e02-4190-b15b-f64a1fc59bbc-ca-trust-extracted\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.141035 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.140815 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2v8lh\" (UniqueName: \"kubernetes.io/projected/8269cb04-9e02-4190-b15b-f64a1fc59bbc-kube-api-access-2v8lh\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:25:38.292678 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.292592 2563 generic.go:356] "Generic (PLEG): container finished" podID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" containerID="0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc" exitCode=0 Jul 06 16:25:38.292678 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.292650 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" Jul 06 16:25:38.292678 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.292652 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" event={"ID":"8269cb04-9e02-4190-b15b-f64a1fc59bbc","Type":"ContainerDied","Data":"0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc"} Jul 06 16:25:38.292985 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.292731 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-fc8cdfbd7-8k66g" event={"ID":"8269cb04-9e02-4190-b15b-f64a1fc59bbc","Type":"ContainerDied","Data":"059588c4f4e75509a406ca59f0e8c9fe553eb5122412548f1a87f65ed2fc9a47"} Jul 06 16:25:38.292985 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.292757 2563 scope.go:117] "RemoveContainer" containerID="0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc" Jul 06 16:25:38.301446 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.301429 2563 scope.go:117] "RemoveContainer" containerID="0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc" Jul 06 16:25:38.301751 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:25:38.301724 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc\": container with ID starting with 0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc not found: ID does not exist" containerID="0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc" Jul 06 16:25:38.301832 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.301759 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc"} err="failed to get container status \"0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc\": rpc error: code = NotFound desc = could not find container \"0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc\": container with ID starting with 0ada468080ba2f2112f0acaa4b81516e3a961f13f81814a01e85c21d8ae236bc not found: ID does not exist" Jul 06 16:25:38.324434 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.324411 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-fc8cdfbd7-8k66g"] Jul 06 16:25:38.339199 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:38.339179 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-fc8cdfbd7-8k66g"] Jul 06 16:25:39.792293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:25:39.792254 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" path="/var/lib/kubelet/pods/8269cb04-9e02-4190-b15b-f64a1fc59bbc/volumes" Jul 06 16:27:53.397620 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.397588 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc"] Jul 06 16:27:53.398130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.397876 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" containerName="registry" Jul 06 16:27:53.398130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.397887 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" containerName="registry" Jul 06 16:27:53.398130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.397938 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8269cb04-9e02-4190-b15b-f64a1fc59bbc" containerName="registry" Jul 06 16:27:53.400426 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.400412 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.404536 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.404513 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 06 16:27:53.404780 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.404767 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 06 16:27:53.404911 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.404890 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-g2mrg\"" Jul 06 16:27:53.415922 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.415900 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc"] Jul 06 16:27:53.523867 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.523830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w9hf\" (UniqueName: \"kubernetes.io/projected/372419e6-b086-4d68-b216-0fcbfa7b8cb1-kube-api-access-2w9hf\") pod \"cert-manager-operator-controller-manager-55c444bf86-c9bqc\" (UID: \"372419e6-b086-4d68-b216-0fcbfa7b8cb1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.524052 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.523896 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/372419e6-b086-4d68-b216-0fcbfa7b8cb1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c9bqc\" (UID: \"372419e6-b086-4d68-b216-0fcbfa7b8cb1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.624579 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.624543 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/372419e6-b086-4d68-b216-0fcbfa7b8cb1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c9bqc\" (UID: \"372419e6-b086-4d68-b216-0fcbfa7b8cb1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.624765 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.624589 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w9hf\" (UniqueName: \"kubernetes.io/projected/372419e6-b086-4d68-b216-0fcbfa7b8cb1-kube-api-access-2w9hf\") pod \"cert-manager-operator-controller-manager-55c444bf86-c9bqc\" (UID: \"372419e6-b086-4d68-b216-0fcbfa7b8cb1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.625012 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.624988 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/372419e6-b086-4d68-b216-0fcbfa7b8cb1-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-c9bqc\" (UID: \"372419e6-b086-4d68-b216-0fcbfa7b8cb1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.640935 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.640904 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w9hf\" (UniqueName: \"kubernetes.io/projected/372419e6-b086-4d68-b216-0fcbfa7b8cb1-kube-api-access-2w9hf\") pod \"cert-manager-operator-controller-manager-55c444bf86-c9bqc\" (UID: \"372419e6-b086-4d68-b216-0fcbfa7b8cb1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.709901 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.709820 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" Jul 06 16:27:53.870542 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:53.870503 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc"] Jul 06 16:27:53.875883 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:27:53.875852 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372419e6_b086_4d68_b216_0fcbfa7b8cb1.slice/crio-12b0b5fc3b14c608a69942a923f44db446914f5a717220109b6fe887a3a1fb3b WatchSource:0}: Error finding container 12b0b5fc3b14c608a69942a923f44db446914f5a717220109b6fe887a3a1fb3b: Status 404 returned error can't find the container with id 12b0b5fc3b14c608a69942a923f44db446914f5a717220109b6fe887a3a1fb3b Jul 06 16:27:54.764993 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:54.764953 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" event={"ID":"372419e6-b086-4d68-b216-0fcbfa7b8cb1","Type":"ContainerStarted","Data":"12b0b5fc3b14c608a69942a923f44db446914f5a717220109b6fe887a3a1fb3b"} Jul 06 16:27:56.773694 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:56.773614 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" event={"ID":"372419e6-b086-4d68-b216-0fcbfa7b8cb1","Type":"ContainerStarted","Data":"7df6946f173b62439566fe75846385470bfa070b9d341eb12d9801606a008163"} Jul 06 16:27:56.804619 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:27:56.804574 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-c9bqc" podStartSLOduration=1.172857422 podStartE2EDuration="3.804560577s" totalImagesPullingTime="2.631703155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:27:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:27:56.802720683 +0000 UTC m=+259.517939666" watchObservedRunningTime="2026-07-06 16:27:56.804560577 +0000 UTC m=+259.519779551" Jul 06 16:28:30.184922 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.184890 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv"] Jul 06 16:28:30.187260 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.187243 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.192424 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.192394 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w774n\" (UniqueName: \"kubernetes.io/projected/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-kube-api-access-w774n\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.192538 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.192426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-manifests\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.192538 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.192450 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.192538 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.192494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-tmp\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.192675 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.192562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.193329 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.193305 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 06 16:28:30.193440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.193341 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 06 16:28:30.193440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.193384 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 06 16:28:30.194027 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.194009 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lbdzb\"" Jul 06 16:28:30.194108 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.194011 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 06 16:28:30.227464 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.227433 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv"] Jul 06 16:28:30.292982 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.292947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.293152 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.293025 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w774n\" (UniqueName: \"kubernetes.io/projected/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-kube-api-access-w774n\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.293152 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.293058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-manifests\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.293152 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.293080 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.293152 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.293108 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-tmp\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.293581 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.293552 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-manifests\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.295427 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.295395 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-tmp\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.295564 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.295544 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.295605 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.295550 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-webhook-cert\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.328174 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.328148 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w774n\" (UniqueName: \"kubernetes.io/projected/d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2-kube-api-access-w774n\") pod \"opendatahub-operator-controller-manager-7896b945bc-mjhzv\" (UID: \"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2\") " pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.497136 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.497041 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:30.682507 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.682479 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv"] Jul 06 16:28:30.684939 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:28:30.684913 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b5b93d_5d4b_42a1_b3bc_2f5ae4bde4b2.slice/crio-fe2ae86fa8774a45788958d58b6446ac79212f0973b97eccb60a9cd6681f8df1 WatchSource:0}: Error finding container fe2ae86fa8774a45788958d58b6446ac79212f0973b97eccb60a9cd6681f8df1: Status 404 returned error can't find the container with id fe2ae86fa8774a45788958d58b6446ac79212f0973b97eccb60a9cd6681f8df1 Jul 06 16:28:30.899836 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:30.899771 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" event={"ID":"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2","Type":"ContainerStarted","Data":"fe2ae86fa8774a45788958d58b6446ac79212f0973b97eccb60a9cd6681f8df1"} Jul 06 16:28:34.918162 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:34.918126 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" event={"ID":"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2","Type":"ContainerStarted","Data":"0c46a1a696b927b3dd00a79af39a65957a955fd07d787df3aaeba7c7f94da1e9"} Jul 06 16:28:35.922709 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:35.922675 2563 generic.go:356] "Generic (PLEG): container finished" podID="d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2" containerID="0c46a1a696b927b3dd00a79af39a65957a955fd07d787df3aaeba7c7f94da1e9" exitCode=0 Jul 06 16:28:35.923098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:35.922718 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" event={"ID":"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2","Type":"ContainerDied","Data":"0c46a1a696b927b3dd00a79af39a65957a955fd07d787df3aaeba7c7f94da1e9"} Jul 06 16:28:36.932723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:36.932683 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" event={"ID":"d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2","Type":"ContainerStarted","Data":"9e4627195c6a859f33866ec15cddbcbf9b79eb4c8df84a6892850d9446e6a61c"} Jul 06 16:28:36.933120 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:36.932833 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:37.694172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:37.694145 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:28:37.696499 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:37.696476 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:28:37.697033 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:37.697016 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 06 16:28:47.938675 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:47.938643 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" Jul 06 16:28:48.030290 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:47.972169 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7896b945bc-mjhzv" podStartSLOduration=13.204235567 podStartE2EDuration="17.972152232s" totalImagesPullingTime="3.742673098s" totalInitContainerRuntime="1.025243567s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:37.009647959 +0000 UTC m=+299.724866948" watchObservedRunningTime="2026-07-06 16:28:47.972152232 +0000 UTC m=+310.687371205" Jul 06 16:28:50.662502 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.662471 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-62n4t"] Jul 06 16:28:50.664734 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.664718 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:50.669908 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.669872 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 06 16:28:50.671089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.670850 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-6pmsn\"" Jul 06 16:28:50.680569 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.680542 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-62n4t"] Jul 06 16:28:50.748224 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.748189 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb2v4\" (UniqueName: \"kubernetes.io/projected/70cb66d3-9dc3-464d-8b61-4945c1270a9d-kube-api-access-mb2v4\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:50.748383 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.748235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb66d3-9dc3-464d-8b61-4945c1270a9d-cert\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:50.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.849273 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb66d3-9dc3-464d-8b61-4945c1270a9d-cert\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:50.849460 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:28:50.849421 2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 06 16:28:50.849460 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.849435 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb2v4\" (UniqueName: \"kubernetes.io/projected/70cb66d3-9dc3-464d-8b61-4945c1270a9d-kube-api-access-mb2v4\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:50.849544 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:28:50.849490 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70cb66d3-9dc3-464d-8b61-4945c1270a9d-cert podName:70cb66d3-9dc3-464d-8b61-4945c1270a9d nodeName:}" failed. No retries permitted until 2026-07-06 16:28:51.349472717 +0000 UTC m=+314.064691669 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/70cb66d3-9dc3-464d-8b61-4945c1270a9d-cert") pod "odh-model-controller-7998c67dc4-62n4t" (UID: "70cb66d3-9dc3-464d-8b61-4945c1270a9d") : secret "odh-model-controller-webhook-cert" not found Jul 06 16:28:50.866166 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:50.866141 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb2v4\" (UniqueName: \"kubernetes.io/projected/70cb66d3-9dc3-464d-8b61-4945c1270a9d-kube-api-access-mb2v4\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:51.353989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.353957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb66d3-9dc3-464d-8b61-4945c1270a9d-cert\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:51.356431 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.356403 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/70cb66d3-9dc3-464d-8b61-4945c1270a9d-cert\") pod \"odh-model-controller-7998c67dc4-62n4t\" (UID: \"70cb66d3-9dc3-464d-8b61-4945c1270a9d\") " pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:51.398300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.398272 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p8tjw"] Jul 06 16:28:51.400854 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.400836 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.405225 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.405201 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 06 16:28:51.431741 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.431708 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p8tjw"] Jul 06 16:28:51.454761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.454734 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n992x\" (UniqueName: \"kubernetes.io/projected/39b64fcc-6538-4001-9c0e-f426eef87f30-kube-api-access-n992x\") pod \"model-serving-api-54d89bd5fc-p8tjw\" (UID: \"39b64fcc-6538-4001-9c0e-f426eef87f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.454907 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.454780 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b64fcc-6538-4001-9c0e-f426eef87f30-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8tjw\" (UID: \"39b64fcc-6538-4001-9c0e-f426eef87f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.555438 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.555406 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b64fcc-6538-4001-9c0e-f426eef87f30-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8tjw\" (UID: \"39b64fcc-6538-4001-9c0e-f426eef87f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.555593 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.555467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n992x\" (UniqueName: \"kubernetes.io/projected/39b64fcc-6538-4001-9c0e-f426eef87f30-kube-api-access-n992x\") pod \"model-serving-api-54d89bd5fc-p8tjw\" (UID: \"39b64fcc-6538-4001-9c0e-f426eef87f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.557973 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.557951 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/39b64fcc-6538-4001-9c0e-f426eef87f30-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8tjw\" (UID: \"39b64fcc-6538-4001-9c0e-f426eef87f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.565054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.565027 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n992x\" (UniqueName: \"kubernetes.io/projected/39b64fcc-6538-4001-9c0e-f426eef87f30-kube-api-access-n992x\") pod \"model-serving-api-54d89bd5fc-p8tjw\" (UID: \"39b64fcc-6538-4001-9c0e-f426eef87f30\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.574866 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.574839 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:51.703144 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.702997 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-62n4t"] Jul 06 16:28:51.705117 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:28:51.705092 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70cb66d3_9dc3_464d_8b61_4945c1270a9d.slice/crio-10c2685f747199af169866bc2ea6b272a2a93466cdd20001f931b3ed52ac68f6 WatchSource:0}: Error finding container 10c2685f747199af169866bc2ea6b272a2a93466cdd20001f931b3ed52ac68f6: Status 404 returned error can't find the container with id 10c2685f747199af169866bc2ea6b272a2a93466cdd20001f931b3ed52ac68f6 Jul 06 16:28:51.706413 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.706399 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:28:51.710661 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.710645 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:51.836931 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.836902 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p8tjw"] Jul 06 16:28:51.991761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.991674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" event={"ID":"70cb66d3-9dc3-464d-8b61-4945c1270a9d","Type":"ContainerStarted","Data":"10c2685f747199af169866bc2ea6b272a2a93466cdd20001f931b3ed52ac68f6"} Jul 06 16:28:51.992691 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:51.992665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" event={"ID":"39b64fcc-6538-4001-9c0e-f426eef87f30","Type":"ContainerStarted","Data":"06ee815bbb7cca0f49b574f142fc30e629160231d38397372e639f81d6333564"} Jul 06 16:28:52.075183 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.072984 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq"] Jul 06 16:28:52.077262 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.077233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.080436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.080411 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 06 16:28:52.080436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.080424 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 06 16:28:52.080594 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.080494 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nr4wj\"" Jul 06 16:28:52.090820 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.090784 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq"] Jul 06 16:28:52.160690 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.160655 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63fed56d-b091-4741-8cdc-5e47db6ed80d-tmp\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.160884 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.160696 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fed56d-b091-4741-8cdc-5e47db6ed80d-tls-certs\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.160884 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.160762 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h4lz\" (UniqueName: \"kubernetes.io/projected/63fed56d-b091-4741-8cdc-5e47db6ed80d-kube-api-access-7h4lz\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.262045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.261965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63fed56d-b091-4741-8cdc-5e47db6ed80d-tmp\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.262045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.262000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fed56d-b091-4741-8cdc-5e47db6ed80d-tls-certs\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.262045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.262038 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h4lz\" (UniqueName: \"kubernetes.io/projected/63fed56d-b091-4741-8cdc-5e47db6ed80d-kube-api-access-7h4lz\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.264294 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.264270 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63fed56d-b091-4741-8cdc-5e47db6ed80d-tmp\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.264482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.264459 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/63fed56d-b091-4741-8cdc-5e47db6ed80d-tls-certs\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.272197 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.272173 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h4lz\" (UniqueName: \"kubernetes.io/projected/63fed56d-b091-4741-8cdc-5e47db6ed80d-kube-api-access-7h4lz\") pod \"kube-auth-proxy-5846db9b4f-nkmxq\" (UID: \"63fed56d-b091-4741-8cdc-5e47db6ed80d\") " pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.386678 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.386643 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" Jul 06 16:28:52.530037 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:52.529960 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq"] Jul 06 16:28:52.533315 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:28:52.533290 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63fed56d_b091_4741_8cdc_5e47db6ed80d.slice/crio-ee1eb04a2ac189a43fb637bd8a87d606c108082106ddafe23a6379831ac21d12 WatchSource:0}: Error finding container ee1eb04a2ac189a43fb637bd8a87d606c108082106ddafe23a6379831ac21d12: Status 404 returned error can't find the container with id ee1eb04a2ac189a43fb637bd8a87d606c108082106ddafe23a6379831ac21d12 Jul 06 16:28:53.000234 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.000197 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" event={"ID":"63fed56d-b091-4741-8cdc-5e47db6ed80d","Type":"ContainerStarted","Data":"ee1eb04a2ac189a43fb637bd8a87d606c108082106ddafe23a6379831ac21d12"} Jul 06 16:28:53.107745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.107711 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9"] Jul 06 16:28:53.109918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.109895 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.113363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.113337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 06 16:28:53.113363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.113350 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 16:28:53.113570 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.113337 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 06 16:28:53.113570 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.113452 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-cnvcg\"" Jul 06 16:28:53.113680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.113616 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 06 16:28:53.125861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.125823 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9"] Jul 06 16:28:53.171457 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9kq8\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-kube-api-access-q9kq8\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.171457 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171412 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-istio-token\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.171708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171521 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de65593-7420-4ead-ae49-de375a946f8b-local-certs\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.171708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171579 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.171708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171617 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-cacerts\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.171708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171638 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.171708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.171687 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.272919 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-istio-token\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.273039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de65593-7420-4ead-ae49-de375a946f8b-local-certs\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.273083 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.273119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-cacerts\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.273147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.273201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.274588 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.273230 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9kq8\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-kube-api-access-q9kq8\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.275423 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.274892 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.279357 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.279326 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.281236 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.281190 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de65593-7420-4ead-ae49-de375a946f8b-local-certs\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.281364 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.281341 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-cacerts\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.285330 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.285285 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.294307 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.294276 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9kq8\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-kube-api-access-q9kq8\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.295547 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.295503 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-istio-token\") pod \"istiod-openshift-gateway-7574459656-7vrc9\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:53.422435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:53.422392 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:28:54.443571 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:54.443541 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9"] Jul 06 16:28:54.446483 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:28:54.446458 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de65593_7420_4ead_ae49_de375a946f8b.slice/crio-6fc118e6ade2394745ea12e68a56152a0e019285eb4c5b1fa7f873654d1e613d WatchSource:0}: Error finding container 6fc118e6ade2394745ea12e68a56152a0e019285eb4c5b1fa7f873654d1e613d: Status 404 returned error can't find the container with id 6fc118e6ade2394745ea12e68a56152a0e019285eb4c5b1fa7f873654d1e613d Jul 06 16:28:55.013379 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:55.013336 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" event={"ID":"1de65593-7420-4ead-ae49-de375a946f8b","Type":"ContainerStarted","Data":"6fc118e6ade2394745ea12e68a56152a0e019285eb4c5b1fa7f873654d1e613d"} Jul 06 16:28:55.015410 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:55.015278 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" event={"ID":"39b64fcc-6538-4001-9c0e-f426eef87f30","Type":"ContainerStarted","Data":"102fad640134d4367084a12e52c8634edbdddcc2556d8e0a1542c35d87b77fa1"} Jul 06 16:28:55.015553 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:55.015484 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:28:55.036778 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:55.036729 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" podStartSLOduration=1.5647377489999998 podStartE2EDuration="4.036709454s" totalImagesPullingTime="2.471971705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:55.036152905 +0000 UTC m=+317.751371880" watchObservedRunningTime="2026-07-06 16:28:55.036709454 +0000 UTC m=+317.751928429" Jul 06 16:28:56.021665 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:56.021627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" event={"ID":"70cb66d3-9dc3-464d-8b61-4945c1270a9d","Type":"ContainerStarted","Data":"3740194aedf28231895b24661d59de6c2eee21558c3866588ca4b5c478e9daa1"} Jul 06 16:28:56.022105 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:56.021892 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:28:56.048141 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:56.048084 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" podStartSLOduration=2.400464992 podStartE2EDuration="6.048064386s" totalImagesPullingTime="3.647599394s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:56.047887906 +0000 UTC m=+318.763106903" watchObservedRunningTime="2026-07-06 16:28:56.048064386 +0000 UTC m=+318.763283359" Jul 06 16:28:57.027119 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:57.027025 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" event={"ID":"63fed56d-b091-4741-8cdc-5e47db6ed80d","Type":"ContainerStarted","Data":"0c8198cc93a7e93a70f43abb7c4dca326d57211b7dd97940d0e2e7a3a4dff917"} Jul 06 16:28:57.056507 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:28:57.056433 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5846db9b4f-nkmxq" podStartSLOduration=0.988843105 podStartE2EDuration="5.056413792s" totalImagesPullingTime="4.067570687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:28:57.055525583 +0000 UTC m=+319.770744558" watchObservedRunningTime="2026-07-06 16:28:57.056413792 +0000 UTC m=+319.771632767" Jul 06 16:29:01.217085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:01.217043 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:29:01.217348 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:01.217116 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:29:02.050723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:02.050693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" event={"ID":"1de65593-7420-4ead-ae49-de375a946f8b","Type":"ContainerStarted","Data":"e5148ac22c17f72fc11a940112a7187e790e152b982851a6280dc07fa9a075a9"} Jul 06 16:29:02.050924 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:02.050851 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:29:02.052383 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:02.052357 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-7vrc9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:29:02.052494 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:02.052408 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:29:03.055978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:03.055947 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-7vrc9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:29:03.056349 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:03.055999 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:29:06.026512 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:06.026484 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p8tjw" Jul 06 16:29:06.056723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:06.056695 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:29:06.442282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:06.442222 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" podStartSLOduration=6.674440881 podStartE2EDuration="13.442204453s" totalImagesPullingTime="6.767763572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:28:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:03.905833649 +0000 UTC m=+326.621052639" watchObservedRunningTime="2026-07-06 16:29:06.442204453 +0000 UTC m=+329.157423429" Jul 06 16:29:07.029770 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:07.029740 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-62n4t" Jul 06 16:29:08.904148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:08.904117 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9"] Jul 06 16:29:08.904613 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:08.904340 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" containerID="cri-o://e5148ac22c17f72fc11a940112a7187e790e152b982851a6280dc07fa9a075a9" gracePeriod=30 Jul 06 16:29:09.055683 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.055648 2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-7vrc9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.22:8080/ready\": dial tcp 10.133.0.22:8080: connect: connection refused" start-of-body= Jul 06 16:29:09.055875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.055738 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" probeResult="failure" output="Get \"http://10.133.0.22:8080/ready\": dial tcp 10.133.0.22:8080: connect: connection refused" Jul 06 16:29:09.081450 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.081411 2563 generic.go:356] "Generic (PLEG): container finished" podID="1de65593-7420-4ead-ae49-de375a946f8b" containerID="e5148ac22c17f72fc11a940112a7187e790e152b982851a6280dc07fa9a075a9" exitCode=0 Jul 06 16:29:09.081601 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.081476 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" event={"ID":"1de65593-7420-4ead-ae49-de375a946f8b","Type":"ContainerDied","Data":"e5148ac22c17f72fc11a940112a7187e790e152b982851a6280dc07fa9a075a9"} Jul 06 16:29:09.142153 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.142130 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:29:09.210142 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210068 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-istio-token\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210142 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210105 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-ca-configmap\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210142 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210140 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-dns-cert\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210395 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210191 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9kq8\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-kube-api-access-q9kq8\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210395 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210285 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-kubeconfig\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210509 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210396 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de65593-7420-4ead-ae49-de375a946f8b-local-certs\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210509 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210436 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-cacerts\") pod \"1de65593-7420-4ead-ae49-de375a946f8b\" (UID: \"1de65593-7420-4ead-ae49-de375a946f8b\") " Jul 06 16:29:09.210615 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210570 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:29:09.210690 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.210673 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-ca-configmap\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:09.213242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.213199 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de65593-7420-4ead-ae49-de375a946f8b-local-certs" (OuterVolumeSpecName: "local-certs") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:29:09.213242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.213199 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-kube-api-access-q9kq8" (OuterVolumeSpecName: "kube-api-access-q9kq8") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "kube-api-access-q9kq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:29:09.213443 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.213387 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-istio-token" (OuterVolumeSpecName: "istio-token") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:29:09.213485 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.213454 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:29:09.213772 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.213749 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:29:09.213879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.213762 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-cacerts" (OuterVolumeSpecName: "cacerts") pod "1de65593-7420-4ead-ae49-de375a946f8b" (UID: "1de65593-7420-4ead-ae49-de375a946f8b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:29:09.311354 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.311309 2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de65593-7420-4ead-ae49-de375a946f8b-local-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:09.311354 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.311344 2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-cacerts\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:09.311354 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.311353 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:09.311354 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.311363 2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-csr-dns-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:09.311623 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.311372 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9kq8\" (UniqueName: \"kubernetes.io/projected/1de65593-7420-4ead-ae49-de375a946f8b-kube-api-access-q9kq8\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:09.311623 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:09.311382 2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1de65593-7420-4ead-ae49-de375a946f8b-istio-kubeconfig\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:29:10.086851 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:10.086822 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" Jul 06 16:29:10.086851 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:10.086841 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9" event={"ID":"1de65593-7420-4ead-ae49-de375a946f8b","Type":"ContainerDied","Data":"6fc118e6ade2394745ea12e68a56152a0e019285eb4c5b1fa7f873654d1e613d"} Jul 06 16:29:10.087328 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:10.086903 2563 scope.go:117] "RemoveContainer" containerID="e5148ac22c17f72fc11a940112a7187e790e152b982851a6280dc07fa9a075a9" Jul 06 16:29:10.157146 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:10.157110 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9"] Jul 06 16:29:10.165068 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:10.165049 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-7vrc9"] Jul 06 16:29:11.792536 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:11.792498 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1de65593-7420-4ead-ae49-de375a946f8b" path="/var/lib/kubelet/pods/1de65593-7420-4ead-ae49-de375a946f8b/volumes" Jul 06 16:29:13.625507 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.625464 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4"] Jul 06 16:29:13.626009 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.625752 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" Jul 06 16:29:13.626009 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.625763 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" Jul 06 16:29:13.626009 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.625837 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1de65593-7420-4ead-ae49-de375a946f8b" containerName="discovery" Jul 06 16:29:13.628643 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.628623 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp"] Jul 06 16:29:13.628787 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.628768 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.630738 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.630720 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh"] Jul 06 16:29:13.630875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.630861 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.632466 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.632451 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.633031 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.633012 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 06 16:29:13.633133 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.633053 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9zd4l\"" Jul 06 16:29:13.662185 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.662163 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 06 16:29:13.662185 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.662165 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gl82v\"" Jul 06 16:29:13.662372 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.662172 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 06 16:29:13.662444 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.662426 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-2c7xw\"" Jul 06 16:29:13.663775 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.663746 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp"] Jul 06 16:29:13.665044 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.665030 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4"] Jul 06 16:29:13.666292 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.666273 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh"] Jul 06 16:29:13.747252 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747218 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747252 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747249 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747280 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566106b0-3f8d-453b-8c41-7f21e1154b70-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8plkp\" (UniqueName: \"kubernetes.io/projected/566106b0-3f8d-453b-8c41-7f21e1154b70-kube-api-access-8plkp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747369 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef614215-9aed-4c52-b811-3c5612cde758-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-47ctp\" (UID: \"ef614215-9aed-4c52-b811-3c5612cde758\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.747478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747389 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747439 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rk9\" (UniqueName: \"kubernetes.io/projected/b3b81d55-e8b2-4502-b8dc-6844ea4289f7-kube-api-access-k9rk9\") pod \"kserve-controller-manager-d6f4b65b8-kf4b4\" (UID: \"b3b81d55-e8b2-4502-b8dc-6844ea4289f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.747774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb8jq\" (UniqueName: \"kubernetes.io/projected/ef614215-9aed-4c52-b811-3c5612cde758-kube-api-access-gb8jq\") pod \"llmisvc-controller-manager-777c6bd7d5-47ctp\" (UID: \"ef614215-9aed-4c52-b811-3c5612cde758\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.747774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.747774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747636 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3b81d55-e8b2-4502-b8dc-6844ea4289f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-kf4b4\" (UID: \"b3b81d55-e8b2-4502-b8dc-6844ea4289f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.747774 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.747706 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849036 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.848988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849036 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849041 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849060 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849076 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566106b0-3f8d-453b-8c41-7f21e1154b70-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849105 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8plkp\" (UniqueName: \"kubernetes.io/projected/566106b0-3f8d-453b-8c41-7f21e1154b70-kube-api-access-8plkp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849125 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849145 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef614215-9aed-4c52-b811-3c5612cde758-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-47ctp\" (UID: \"ef614215-9aed-4c52-b811-3c5612cde758\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849171 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849198 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849259 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rk9\" (UniqueName: \"kubernetes.io/projected/b3b81d55-e8b2-4502-b8dc-6844ea4289f7-kube-api-access-k9rk9\") pod \"kserve-controller-manager-d6f4b65b8-kf4b4\" (UID: \"b3b81d55-e8b2-4502-b8dc-6844ea4289f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.849301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb8jq\" (UniqueName: \"kubernetes.io/projected/ef614215-9aed-4c52-b811-3c5612cde758-kube-api-access-gb8jq\") pod \"llmisvc-controller-manager-777c6bd7d5-47ctp\" (UID: \"ef614215-9aed-4c52-b811-3c5612cde758\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.849745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849313 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3b81d55-e8b2-4502-b8dc-6844ea4289f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-kf4b4\" (UID: \"b3b81d55-e8b2-4502-b8dc-6844ea4289f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.849745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849506 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849618 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-data\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849741 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849986 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849848 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.849986 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.849942 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566106b0-3f8d-453b-8c41-7f21e1154b70-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.851557 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.851533 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.851768 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.851751 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3b81d55-e8b2-4502-b8dc-6844ea4289f7-cert\") pod \"kserve-controller-manager-d6f4b65b8-kf4b4\" (UID: \"b3b81d55-e8b2-4502-b8dc-6844ea4289f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.851859 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.851766 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ef614215-9aed-4c52-b811-3c5612cde758-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-47ctp\" (UID: \"ef614215-9aed-4c52-b811-3c5612cde758\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.852018 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.852002 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.860438 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.860407 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566106b0-3f8d-453b-8c41-7f21e1154b70-istio-token\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.860604 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.860588 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb8jq\" (UniqueName: \"kubernetes.io/projected/ef614215-9aed-4c52-b811-3c5612cde758-kube-api-access-gb8jq\") pod \"llmisvc-controller-manager-777c6bd7d5-47ctp\" (UID: \"ef614215-9aed-4c52-b811-3c5612cde758\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.860784 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.860755 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rk9\" (UniqueName: \"kubernetes.io/projected/b3b81d55-e8b2-4502-b8dc-6844ea4289f7-kube-api-access-k9rk9\") pod \"kserve-controller-manager-d6f4b65b8-kf4b4\" (UID: \"b3b81d55-e8b2-4502-b8dc-6844ea4289f7\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.860964 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.860949 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8plkp\" (UniqueName: \"kubernetes.io/projected/566106b0-3f8d-453b-8c41-7f21e1154b70-kube-api-access-8plkp\") pod \"data-science-gateway-data-science-gateway-class-7f964fb678f8grh\" (UID: \"566106b0-3f8d-453b-8c41-7f21e1154b70\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:13.939932 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.939822 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:13.946644 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.946624 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:13.953782 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:13.953615 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:14.086595 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:14.086571 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp"] Jul 06 16:29:14.089526 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:29:14.089486 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef614215_9aed_4c52_b811_3c5612cde758.slice/crio-62929f9238fd93b807761174724e46021c94043eca0bf196fc91c91ad0e41cdd WatchSource:0}: Error finding container 62929f9238fd93b807761174724e46021c94043eca0bf196fc91c91ad0e41cdd: Status 404 returned error can't find the container with id 62929f9238fd93b807761174724e46021c94043eca0bf196fc91c91ad0e41cdd Jul 06 16:29:14.104540 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:14.104512 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" event={"ID":"ef614215-9aed-4c52-b811-3c5612cde758","Type":"ContainerStarted","Data":"62929f9238fd93b807761174724e46021c94043eca0bf196fc91c91ad0e41cdd"} Jul 06 16:29:14.316727 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:14.316705 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4"] Jul 06 16:29:14.318717 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:29:14.318687 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3b81d55_e8b2_4502_b8dc_6844ea4289f7.slice/crio-7d2ae37425cb58974c427d1f421698fb127575c61865133891aa74627a4f6f89 WatchSource:0}: Error finding container 7d2ae37425cb58974c427d1f421698fb127575c61865133891aa74627a4f6f89: Status 404 returned error can't find the container with id 7d2ae37425cb58974c427d1f421698fb127575c61865133891aa74627a4f6f89 Jul 06 16:29:14.322408 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:14.322361 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh"] Jul 06 16:29:14.324335 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:29:14.324311 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566106b0_3f8d_453b_8c41_7f21e1154b70.slice/crio-9d9bb1cae5d2aec25ecf7f29c222250520c65eb9f22f270f4676884f7cf9b5c9 WatchSource:0}: Error finding container 9d9bb1cae5d2aec25ecf7f29c222250520c65eb9f22f270f4676884f7cf9b5c9: Status 404 returned error can't find the container with id 9d9bb1cae5d2aec25ecf7f29c222250520c65eb9f22f270f4676884f7cf9b5c9 Jul 06 16:29:15.111578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:15.111532 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" event={"ID":"b3b81d55-e8b2-4502-b8dc-6844ea4289f7","Type":"ContainerStarted","Data":"7d2ae37425cb58974c427d1f421698fb127575c61865133891aa74627a4f6f89"} Jul 06 16:29:15.112997 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:15.112958 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" event={"ID":"566106b0-3f8d-453b-8c41-7f21e1154b70","Type":"ContainerStarted","Data":"9d9bb1cae5d2aec25ecf7f29c222250520c65eb9f22f270f4676884f7cf9b5c9"} Jul 06 16:29:17.638163 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:17.638113 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:29:17.638532 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:17.638222 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:29:17.638532 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:17.638275 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:29:18.130253 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.130222 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" event={"ID":"ef614215-9aed-4c52-b811-3c5612cde758","Type":"ContainerStarted","Data":"db64d6701adc25b7b2a5c1ebf1806567ad5227241f7ec06a70f95371b55454e3"} Jul 06 16:29:18.130435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.130297 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:18.132136 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.132108 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" event={"ID":"566106b0-3f8d-453b-8c41-7f21e1154b70","Type":"ContainerStarted","Data":"2ce13227d4e80aa9d7f30d7faf50a4006b315c839c99ab93369f519d39a4b113"} Jul 06 16:29:18.156915 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.156857 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" podStartSLOduration=1.609316427 podStartE2EDuration="5.156836844s" totalImagesPullingTime="3.547520417s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:18.156069628 +0000 UTC m=+340.871288630" watchObservedRunningTime="2026-07-06 16:29:18.156836844 +0000 UTC m=+340.872055871" Jul 06 16:29:18.192040 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.191979 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" podStartSLOduration=1.8805075709999999 podStartE2EDuration="5.191961583s" totalImagesPullingTime="3.311454012s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:18.191368276 +0000 UTC m=+340.906587287" watchObservedRunningTime="2026-07-06 16:29:18.191961583 +0000 UTC m=+340.907180561" Jul 06 16:29:18.954431 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.954377 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:18.958891 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:18.958866 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:19.138532 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:19.138493 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" event={"ID":"b3b81d55-e8b2-4502-b8dc-6844ea4289f7","Type":"ContainerStarted","Data":"712d6fa0a2ffd82e3cb166cd2a4d550dc5b6f0de33308280bc18cae72cae0fca"} Jul 06 16:29:19.138749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:19.138729 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:29:19.139023 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:19.138997 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:19.140091 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:19.140076 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-7f964fb678f8grh" Jul 06 16:29:19.168425 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:19.168377 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" podStartSLOduration=2.162130936 podStartE2EDuration="6.168357364s" totalImagesPullingTime="4.006226428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:29:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:29:19.166626619 +0000 UTC m=+341.881845606" watchObservedRunningTime="2026-07-06 16:29:19.168357364 +0000 UTC m=+341.883576337" Jul 06 16:29:49.140949 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:49.140916 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-47ctp" Jul 06 16:29:50.148716 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:29:50.148688 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kf4b4" Jul 06 16:30:11.435719 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.435681 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-986ft"] Jul 06 16:30:11.438124 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.438108 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:11.442867 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.442840 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-dp5g9\"" Jul 06 16:30:11.443195 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.442893 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 06 16:30:11.443195 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.442900 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 06 16:30:11.449874 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.449850 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-986ft"] Jul 06 16:30:11.527575 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.527538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfsj6\" (UniqueName: \"kubernetes.io/projected/2358ee01-4ef6-4081-a321-9b987832095c-kube-api-access-hfsj6\") pod \"kuadrant-operator-catalog-986ft\" (UID: \"2358ee01-4ef6-4081-a321-9b987832095c\") " pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:11.628354 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.628315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfsj6\" (UniqueName: \"kubernetes.io/projected/2358ee01-4ef6-4081-a321-9b987832095c-kube-api-access-hfsj6\") pod \"kuadrant-operator-catalog-986ft\" (UID: \"2358ee01-4ef6-4081-a321-9b987832095c\") " pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:11.638132 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.638099 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfsj6\" (UniqueName: \"kubernetes.io/projected/2358ee01-4ef6-4081-a321-9b987832095c-kube-api-access-hfsj6\") pod \"kuadrant-operator-catalog-986ft\" (UID: \"2358ee01-4ef6-4081-a321-9b987832095c\") " pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:11.749160 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.749073 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:11.784856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.784815 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-986ft"] Jul 06 16:30:11.889934 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:11.889905 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-986ft"] Jul 06 16:30:11.892904 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:30:11.892875 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2358ee01_4ef6_4081_a321_9b987832095c.slice/crio-1adb8cc81a7059e30f9e000fcb29a378c6416b71467ad51f696b48b2bdf00f66 WatchSource:0}: Error finding container 1adb8cc81a7059e30f9e000fcb29a378c6416b71467ad51f696b48b2bdf00f66: Status 404 returned error can't find the container with id 1adb8cc81a7059e30f9e000fcb29a378c6416b71467ad51f696b48b2bdf00f66 Jul 06 16:30:12.009286 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.009203 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7lwdw"] Jul 06 16:30:12.012004 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.011989 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:12.023007 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.022984 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7lwdw"] Jul 06 16:30:12.132933 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.132893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2nsb\" (UniqueName: \"kubernetes.io/projected/f054c648-4aa7-4c32-bc3a-e5cb5a04033f-kube-api-access-h2nsb\") pod \"kuadrant-operator-catalog-7lwdw\" (UID: \"f054c648-4aa7-4c32-bc3a-e5cb5a04033f\") " pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:12.234351 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.234301 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2nsb\" (UniqueName: \"kubernetes.io/projected/f054c648-4aa7-4c32-bc3a-e5cb5a04033f-kube-api-access-h2nsb\") pod \"kuadrant-operator-catalog-7lwdw\" (UID: \"f054c648-4aa7-4c32-bc3a-e5cb5a04033f\") " pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:12.245242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.245215 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2nsb\" (UniqueName: \"kubernetes.io/projected/f054c648-4aa7-4c32-bc3a-e5cb5a04033f-kube-api-access-h2nsb\") pod \"kuadrant-operator-catalog-7lwdw\" (UID: \"f054c648-4aa7-4c32-bc3a-e5cb5a04033f\") " pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:12.321959 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.321880 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:12.361299 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.361266 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-986ft" event={"ID":"2358ee01-4ef6-4081-a321-9b987832095c","Type":"ContainerStarted","Data":"1adb8cc81a7059e30f9e000fcb29a378c6416b71467ad51f696b48b2bdf00f66"} Jul 06 16:30:12.448137 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:12.448107 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7lwdw"] Jul 06 16:30:12.450974 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:30:12.450948 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf054c648_4aa7_4c32_bc3a_e5cb5a04033f.slice/crio-d638362e5f58ccfd72d1895f3664b5547d5f5862b20f38643be55fa1d6afec2b WatchSource:0}: Error finding container d638362e5f58ccfd72d1895f3664b5547d5f5862b20f38643be55fa1d6afec2b: Status 404 returned error can't find the container with id d638362e5f58ccfd72d1895f3664b5547d5f5862b20f38643be55fa1d6afec2b Jul 06 16:30:13.368971 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:13.368932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" event={"ID":"f054c648-4aa7-4c32-bc3a-e5cb5a04033f","Type":"ContainerStarted","Data":"d638362e5f58ccfd72d1895f3664b5547d5f5862b20f38643be55fa1d6afec2b"} Jul 06 16:30:15.379522 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.379483 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" event={"ID":"f054c648-4aa7-4c32-bc3a-e5cb5a04033f","Type":"ContainerStarted","Data":"e819c6d45aa28a682b57a5fd9b40bf20709d11e97f0c4a81b8b558548613a60b"} Jul 06 16:30:15.380909 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.380888 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-986ft" event={"ID":"2358ee01-4ef6-4081-a321-9b987832095c","Type":"ContainerStarted","Data":"56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21"} Jul 06 16:30:15.381031 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.380959 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-986ft" podUID="2358ee01-4ef6-4081-a321-9b987832095c" containerName="registry-server" containerID="cri-o://56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21" gracePeriod=2 Jul 06 16:30:15.399276 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.399220 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" podStartSLOduration=2.136900538 podStartE2EDuration="4.399204771s" totalImagesPullingTime="2.262304233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:15.397732396 +0000 UTC m=+398.112951373" watchObservedRunningTime="2026-07-06 16:30:15.399204771 +0000 UTC m=+398.114423745" Jul 06 16:30:15.415560 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.415516 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-986ft" podStartSLOduration=1.5997327019999998 podStartE2EDuration="4.415501472s" totalImagesPullingTime="2.81576877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:15.414913801 +0000 UTC m=+398.130132811" watchObservedRunningTime="2026-07-06 16:30:15.415501472 +0000 UTC m=+398.130720446" Jul 06 16:30:15.624021 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.623996 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:15.764993 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.764903 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfsj6\" (UniqueName: \"kubernetes.io/projected/2358ee01-4ef6-4081-a321-9b987832095c-kube-api-access-hfsj6\") pod \"2358ee01-4ef6-4081-a321-9b987832095c\" (UID: \"2358ee01-4ef6-4081-a321-9b987832095c\") " Jul 06 16:30:15.767067 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.767038 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2358ee01-4ef6-4081-a321-9b987832095c-kube-api-access-hfsj6" (OuterVolumeSpecName: "kube-api-access-hfsj6") pod "2358ee01-4ef6-4081-a321-9b987832095c" (UID: "2358ee01-4ef6-4081-a321-9b987832095c"). InnerVolumeSpecName "kube-api-access-hfsj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:30:15.866244 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:15.866203 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfsj6\" (UniqueName: \"kubernetes.io/projected/2358ee01-4ef6-4081-a321-9b987832095c-kube-api-access-hfsj6\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:30:16.386366 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.386328 2563 generic.go:356] "Generic (PLEG): container finished" podID="2358ee01-4ef6-4081-a321-9b987832095c" containerID="56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21" exitCode=0 Jul 06 16:30:16.386779 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.386387 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-986ft" Jul 06 16:30:16.386779 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.386409 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-986ft" event={"ID":"2358ee01-4ef6-4081-a321-9b987832095c","Type":"ContainerDied","Data":"56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21"} Jul 06 16:30:16.386779 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.386460 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-986ft" event={"ID":"2358ee01-4ef6-4081-a321-9b987832095c","Type":"ContainerDied","Data":"1adb8cc81a7059e30f9e000fcb29a378c6416b71467ad51f696b48b2bdf00f66"} Jul 06 16:30:16.386779 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.386479 2563 scope.go:117] "RemoveContainer" containerID="56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21" Jul 06 16:30:16.394715 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.394695 2563 scope.go:117] "RemoveContainer" containerID="56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21" Jul 06 16:30:16.394991 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:30:16.394972 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21\": container with ID starting with 56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21 not found: ID does not exist" containerID="56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21" Jul 06 16:30:16.395046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.394999 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21"} err="failed to get container status \"56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21\": rpc error: code = NotFound desc = could not find container \"56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21\": container with ID starting with 56033560f20951a70cd99385b81532a09265d41afa09c3b623e612d14acc4d21 not found: ID does not exist" Jul 06 16:30:16.408607 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.408577 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-986ft"] Jul 06 16:30:16.412257 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:16.412230 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-986ft"] Jul 06 16:30:17.792618 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:17.792586 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2358ee01-4ef6-4081-a321-9b987832095c" path="/var/lib/kubelet/pods/2358ee01-4ef6-4081-a321-9b987832095c/volumes" Jul 06 16:30:22.322658 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:22.322618 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:22.323232 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:22.322712 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:22.345393 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:22.345367 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:22.432682 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:22.432653 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7lwdw" Jul 06 16:30:45.892626 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.892591 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt"] Jul 06 16:30:45.893039 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.892919 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2358ee01-4ef6-4081-a321-9b987832095c" containerName="registry-server" Jul 06 16:30:45.893039 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.892933 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2358ee01-4ef6-4081-a321-9b987832095c" containerName="registry-server" Jul 06 16:30:45.893039 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.892990 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2358ee01-4ef6-4081-a321-9b987832095c" containerName="registry-server" Jul 06 16:30:45.896966 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.896946 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:45.900957 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.900921 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n42ft\"" Jul 06 16:30:45.906357 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.906331 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j7pk\" (UniqueName: \"kubernetes.io/projected/ce1c4335-f099-40ec-ae5c-01e27894b3ed-kube-api-access-6j7pk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:45.906503 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.906482 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce1c4335-f099-40ec-ae5c-01e27894b3ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:45.932162 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:45.932138 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt"] Jul 06 16:30:46.006902 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.006870 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce1c4335-f099-40ec-ae5c-01e27894b3ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:46.007050 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.006910 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j7pk\" (UniqueName: \"kubernetes.io/projected/ce1c4335-f099-40ec-ae5c-01e27894b3ed-kube-api-access-6j7pk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:46.007289 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.007269 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce1c4335-f099-40ec-ae5c-01e27894b3ed-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:46.021783 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.021759 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j7pk\" (UniqueName: \"kubernetes.io/projected/ce1c4335-f099-40ec-ae5c-01e27894b3ed-kube-api-access-6j7pk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:46.208056 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.207968 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:46.340395 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.340339 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt"] Jul 06 16:30:46.342676 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:30:46.342652 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1c4335_f099_40ec_ae5c_01e27894b3ed.slice/crio-21f84e8d43a650388e7b0356cdd348e5194b4218c1b056911e6de4abbf6b8a78 WatchSource:0}: Error finding container 21f84e8d43a650388e7b0356cdd348e5194b4218c1b056911e6de4abbf6b8a78: Status 404 returned error can't find the container with id 21f84e8d43a650388e7b0356cdd348e5194b4218c1b056911e6de4abbf6b8a78 Jul 06 16:30:46.511679 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:46.511600 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" event={"ID":"ce1c4335-f099-40ec-ae5c-01e27894b3ed","Type":"ContainerStarted","Data":"21f84e8d43a650388e7b0356cdd348e5194b4218c1b056911e6de4abbf6b8a78"} Jul 06 16:30:48.549827 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.549782 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7"] Jul 06 16:30:48.555678 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.555657 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:30:48.559338 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.559320 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bhq7p\"" Jul 06 16:30:48.576215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.576194 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7"] Jul 06 16:30:48.624933 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.624897 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pbkc\" (UniqueName: \"kubernetes.io/projected/db33abe1-697d-49ff-889e-4ff114f74ce5-kube-api-access-8pbkc\") pod \"authorino-operator-75cc8d5cb7-kqlh7\" (UID: \"db33abe1-697d-49ff-889e-4ff114f74ce5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:30:48.726203 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.726156 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pbkc\" (UniqueName: \"kubernetes.io/projected/db33abe1-697d-49ff-889e-4ff114f74ce5-kube-api-access-8pbkc\") pod \"authorino-operator-75cc8d5cb7-kqlh7\" (UID: \"db33abe1-697d-49ff-889e-4ff114f74ce5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:30:48.751199 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.751166 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pbkc\" (UniqueName: \"kubernetes.io/projected/db33abe1-697d-49ff-889e-4ff114f74ce5-kube-api-access-8pbkc\") pod \"authorino-operator-75cc8d5cb7-kqlh7\" (UID: \"db33abe1-697d-49ff-889e-4ff114f74ce5\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:30:48.866290 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:48.866251 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:30:49.030015 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:49.029988 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7"] Jul 06 16:30:49.529669 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:49.529629 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" event={"ID":"db33abe1-697d-49ff-889e-4ff114f74ce5","Type":"ContainerStarted","Data":"e3033721653578fdf24041b0fdc98c468a197a3d20b8f8c7dba2d3bb64d8b870"} Jul 06 16:30:50.571455 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.571424 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k"] Jul 06 16:30:50.574953 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.574933 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:30:50.578145 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.578125 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-lfqlj\"" Jul 06 16:30:50.588514 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.588493 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k"] Jul 06 16:30:50.642366 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.642326 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6qs\" (UniqueName: \"kubernetes.io/projected/aa698e55-ca60-4956-bf15-e0932e1121dd-kube-api-access-lt6qs\") pod \"limitador-operator-controller-manager-859d7b78bd-xl28k\" (UID: \"aa698e55-ca60-4956-bf15-e0932e1121dd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:30:50.743732 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.743691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6qs\" (UniqueName: \"kubernetes.io/projected/aa698e55-ca60-4956-bf15-e0932e1121dd-kube-api-access-lt6qs\") pod \"limitador-operator-controller-manager-859d7b78bd-xl28k\" (UID: \"aa698e55-ca60-4956-bf15-e0932e1121dd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:30:50.757521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.757364 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6qs\" (UniqueName: \"kubernetes.io/projected/aa698e55-ca60-4956-bf15-e0932e1121dd-kube-api-access-lt6qs\") pod \"limitador-operator-controller-manager-859d7b78bd-xl28k\" (UID: \"aa698e55-ca60-4956-bf15-e0932e1121dd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:30:50.886418 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:50.886382 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:30:51.445823 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:51.445787 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k"] Jul 06 16:30:51.448090 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:30:51.448058 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa698e55_ca60_4956_bf15_e0932e1121dd.slice/crio-a2e339e0bd63cc48140d9d20c34872b3a1d6700254eed2737984fba2d765a74f WatchSource:0}: Error finding container a2e339e0bd63cc48140d9d20c34872b3a1d6700254eed2737984fba2d765a74f: Status 404 returned error can't find the container with id a2e339e0bd63cc48140d9d20c34872b3a1d6700254eed2737984fba2d765a74f Jul 06 16:30:51.541684 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:51.541643 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" event={"ID":"aa698e55-ca60-4956-bf15-e0932e1121dd","Type":"ContainerStarted","Data":"a2e339e0bd63cc48140d9d20c34872b3a1d6700254eed2737984fba2d765a74f"} Jul 06 16:30:53.094565 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.094537 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9"] Jul 06 16:30:53.098102 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.098076 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:30:53.104170 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.104148 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 06 16:30:53.104282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.104193 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-jzjzr\"" Jul 06 16:30:53.123332 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.123307 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9"] Jul 06 16:30:53.164852 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.164826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mcpw\" (UniqueName: \"kubernetes.io/projected/745d1b0e-1e28-4b5d-883e-debffe68b46f-kube-api-access-2mcpw\") pod \"dns-operator-controller-manager-864dd78c4c-4sxp9\" (UID: \"745d1b0e-1e28-4b5d-883e-debffe68b46f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:30:53.265763 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.265725 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mcpw\" (UniqueName: \"kubernetes.io/projected/745d1b0e-1e28-4b5d-883e-debffe68b46f-kube-api-access-2mcpw\") pod \"dns-operator-controller-manager-864dd78c4c-4sxp9\" (UID: \"745d1b0e-1e28-4b5d-883e-debffe68b46f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:30:53.283772 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.283743 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mcpw\" (UniqueName: \"kubernetes.io/projected/745d1b0e-1e28-4b5d-883e-debffe68b46f-kube-api-access-2mcpw\") pod \"dns-operator-controller-manager-864dd78c4c-4sxp9\" (UID: \"745d1b0e-1e28-4b5d-883e-debffe68b46f\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:30:53.423353 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.423321 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:30:53.556629 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.556593 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" event={"ID":"db33abe1-697d-49ff-889e-4ff114f74ce5","Type":"ContainerStarted","Data":"60683cce7c93def5268b1aed377715999b0d4bc2b522bb9c8df9fcb120a31ac8"} Jul 06 16:30:53.556769 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.556722 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:30:53.580012 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.579573 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" podStartSLOduration=1.563330735 podStartE2EDuration="5.579549451s" totalImagesPullingTime="4.016218716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:53.575828815 +0000 UTC m=+436.291047809" watchObservedRunningTime="2026-07-06 16:30:53.579549451 +0000 UTC m=+436.294768407" Jul 06 16:30:53.668231 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:53.668203 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9"] Jul 06 16:30:53.670957 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:30:53.670921 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745d1b0e_1e28_4b5d_883e_debffe68b46f.slice/crio-7824be686f1c5637e6be64ef9809f5a74c6321005b38db8efae419f85778fee3 WatchSource:0}: Error finding container 7824be686f1c5637e6be64ef9809f5a74c6321005b38db8efae419f85778fee3: Status 404 returned error can't find the container with id 7824be686f1c5637e6be64ef9809f5a74c6321005b38db8efae419f85778fee3 Jul 06 16:30:54.564082 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:54.564018 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" event={"ID":"745d1b0e-1e28-4b5d-883e-debffe68b46f","Type":"ContainerStarted","Data":"7824be686f1c5637e6be64ef9809f5a74c6321005b38db8efae419f85778fee3"} Jul 06 16:30:54.566216 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:54.566148 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" event={"ID":"ce1c4335-f099-40ec-ae5c-01e27894b3ed","Type":"ContainerStarted","Data":"58ca68b80d69875bbdb6e89646dea387577c170a4e25d4c4f4ba4183f150b913"} Jul 06 16:30:54.566431 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:54.566411 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:30:54.592908 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:54.592860 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" podStartSLOduration=2.645467761 podStartE2EDuration="9.592845561s" totalImagesPullingTime="6.9473778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:54.589473121 +0000 UTC m=+437.304692153" watchObservedRunningTime="2026-07-06 16:30:54.592845561 +0000 UTC m=+437.308064535" Jul 06 16:30:55.571823 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:55.571752 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" event={"ID":"aa698e55-ca60-4956-bf15-e0932e1121dd","Type":"ContainerStarted","Data":"5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3"} Jul 06 16:30:55.605698 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:55.605645 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" podStartSLOduration=2.428792734 podStartE2EDuration="5.605622974s" totalImagesPullingTime="3.17683024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:55.604061344 +0000 UTC m=+438.319280339" watchObservedRunningTime="2026-07-06 16:30:55.605622974 +0000 UTC m=+438.320841949" Jul 06 16:30:56.578095 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.578062 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" event={"ID":"745d1b0e-1e28-4b5d-883e-debffe68b46f","Type":"ContainerStarted","Data":"e4554b74e46b6dff82982ceab72704b7fc20e86232102efbab8e67082fd43ef7"} Jul 06 16:30:56.578569 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.578152 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:30:56.704499 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.704446 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" podStartSLOduration=1.305240226 podStartE2EDuration="3.704429784s" totalImagesPullingTime="2.399189558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:30:56.601876947 +0000 UTC m=+439.317095942" watchObservedRunningTime="2026-07-06 16:30:56.704429784 +0000 UTC m=+439.419648761" Jul 06 16:30:56.705110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.705090 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k"] Jul 06 16:30:56.708560 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.708540 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.711805 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.711768 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 06 16:30:56.711927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.711846 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-72sgb\"" Jul 06 16:30:56.712112 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.712098 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 06 16:30:56.723079 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.723060 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k"] Jul 06 16:30:56.797605 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.797578 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c6b817-4094-4629-b554-3ef3b9f714cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.797775 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.797612 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbt82\" (UniqueName: \"kubernetes.io/projected/12c6b817-4094-4629-b554-3ef3b9f714cc-kube-api-access-sbt82\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.797775 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.797644 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12c6b817-4094-4629-b554-3ef3b9f714cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.899054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.899016 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c6b817-4094-4629-b554-3ef3b9f714cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.899255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.899221 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbt82\" (UniqueName: \"kubernetes.io/projected/12c6b817-4094-4629-b554-3ef3b9f714cc-kube-api-access-sbt82\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.899413 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.899353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12c6b817-4094-4629-b554-3ef3b9f714cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.900015 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.899995 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/12c6b817-4094-4629-b554-3ef3b9f714cc-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.901461 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.901443 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/12c6b817-4094-4629-b554-3ef3b9f714cc-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:56.915166 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:56.915135 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbt82\" (UniqueName: \"kubernetes.io/projected/12c6b817-4094-4629-b554-3ef3b9f714cc-kube-api-access-sbt82\") pod \"kuadrant-console-plugin-66b5774985-9hp2k\" (UID: \"12c6b817-4094-4629-b554-3ef3b9f714cc\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:57.017675 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:57.017647 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" Jul 06 16:30:57.178508 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:57.178403 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k"] Jul 06 16:30:57.180999 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:30:57.180970 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod12c6b817_4094_4629_b554_3ef3b9f714cc.slice/crio-83a319affc91d8da064ec6ad323706fcb717413cd0e274bba69412d95e49d289 WatchSource:0}: Error finding container 83a319affc91d8da064ec6ad323706fcb717413cd0e274bba69412d95e49d289: Status 404 returned error can't find the container with id 83a319affc91d8da064ec6ad323706fcb717413cd0e274bba69412d95e49d289 Jul 06 16:30:57.584378 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:57.584290 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" event={"ID":"12c6b817-4094-4629-b554-3ef3b9f714cc","Type":"ContainerStarted","Data":"83a319affc91d8da064ec6ad323706fcb717413cd0e274bba69412d95e49d289"} Jul 06 16:30:57.584753 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:30:57.584593 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:31:04.568430 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:04.568341 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-kqlh7" Jul 06 16:31:05.574702 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:05.574670 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:07.586950 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.586915 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:07.698501 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.698471 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt"] Jul 06 16:31:07.698657 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.698638 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" containerName="manager" containerID="cri-o://58ca68b80d69875bbdb6e89646dea387577c170a4e25d4c4f4ba4183f150b913" gracePeriod=2 Jul 06 16:31:07.714160 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.714130 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt"] Jul 06 16:31:07.722687 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.722638 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"ce1c4335-f099-40ec-ae5c-01e27894b3ed\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\": pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:07.731108 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.731081 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k"] Jul 06 16:31:07.731316 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.731275 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" containerName="manager" containerID="cri-o://5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3" gracePeriod=2 Jul 06 16:31:07.735311 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.735123 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:07.738532 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.738490 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k"] Jul 06 16:31:07.739833 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.739773 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xl28k\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:07.744292 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744266 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5"] Jul 06 16:31:07.744778 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744761 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" containerName="manager" Jul 06 16:31:07.744887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744783 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" containerName="manager" Jul 06 16:31:07.744887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744816 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" containerName="manager" Jul 06 16:31:07.744887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744825 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" containerName="manager" Jul 06 16:31:07.745093 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744920 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" containerName="manager" Jul 06 16:31:07.745093 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.744935 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" containerName="manager" Jul 06 16:31:07.747416 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.747393 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.750899 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.750839 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xl28k\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:07.764564 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.764534 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5"] Jul 06 16:31:07.765681 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.765649 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8"] Jul 06 16:31:07.768346 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.768327 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:07.785317 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.785280 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:07.788016 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.787977 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:07.790605 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:07.790570 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xl28k\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:07.795076 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.795054 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8"] Jul 06 16:31:07.797401 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.797374 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d418a4f-b754-4ad3-bae1-78eafcc5555f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfmg5\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.797564 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.797538 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5m5v\" (UniqueName: \"kubernetes.io/projected/9d418a4f-b754-4ad3-bae1-78eafcc5555f-kube-api-access-p5m5v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfmg5\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.898097 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.898011 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5m5v\" (UniqueName: \"kubernetes.io/projected/9d418a4f-b754-4ad3-bae1-78eafcc5555f-kube-api-access-p5m5v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfmg5\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.898097 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.898093 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d418a4f-b754-4ad3-bae1-78eafcc5555f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfmg5\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.898317 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.898172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2rw\" (UniqueName: \"kubernetes.io/projected/91c55469-a3bd-4def-9aaa-7d505c6cf825-kube-api-access-lk2rw\") pod \"limitador-operator-controller-manager-859d7b78bd-tvrm8\" (UID: \"91c55469-a3bd-4def-9aaa-7d505c6cf825\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:07.898632 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.898592 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d418a4f-b754-4ad3-bae1-78eafcc5555f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfmg5\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.919468 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.919392 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5m5v\" (UniqueName: \"kubernetes.io/projected/9d418a4f-b754-4ad3-bae1-78eafcc5555f-kube-api-access-p5m5v\") pod \"kuadrant-operator-controller-manager-8684c9ff49-hfmg5\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:07.999238 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:07.999199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2rw\" (UniqueName: \"kubernetes.io/projected/91c55469-a3bd-4def-9aaa-7d505c6cf825-kube-api-access-lk2rw\") pod \"limitador-operator-controller-manager-859d7b78bd-tvrm8\" (UID: \"91c55469-a3bd-4def-9aaa-7d505c6cf825\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:08.013754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:08.013720 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2rw\" (UniqueName: \"kubernetes.io/projected/91c55469-a3bd-4def-9aaa-7d505c6cf825-kube-api-access-lk2rw\") pod \"limitador-operator-controller-manager-859d7b78bd-tvrm8\" (UID: \"91c55469-a3bd-4def-9aaa-7d505c6cf825\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:08.115742 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:08.115704 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:08.123580 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:08.123548 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:08.592058 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:08.592027 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4sxp9" Jul 06 16:31:08.647938 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:08.647905 2563 generic.go:356] "Generic (PLEG): container finished" podID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" containerID="58ca68b80d69875bbdb6e89646dea387577c170a4e25d4c4f4ba4183f150b913" exitCode=0 Jul 06 16:31:22.026816 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.026589 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8"] Jul 06 16:31:22.031669 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:31:22.031613 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c55469_a3bd_4def_9aaa_7d505c6cf825.slice/crio-e0778c46a52f4f318d8dbab8d0318c37ebda48aec354a84b2a6c18a8764e6a16 WatchSource:0}: Error finding container e0778c46a52f4f318d8dbab8d0318c37ebda48aec354a84b2a6c18a8764e6a16: Status 404 returned error can't find the container with id e0778c46a52f4f318d8dbab8d0318c37ebda48aec354a84b2a6c18a8764e6a16 Jul 06 16:31:22.033964 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.033941 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:22.037597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.037575 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:22.065232 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:31:22.065202 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d418a4f_b754_4ad3_bae1_78eafcc5555f.slice/crio-4775569dcd949e9c70649a1fde5a7dbe3d16ea93fff39b07b995f0290ee47b49 WatchSource:0}: Error finding container 4775569dcd949e9c70649a1fde5a7dbe3d16ea93fff39b07b995f0290ee47b49: Status 404 returned error can't find the container with id 4775569dcd949e9c70649a1fde5a7dbe3d16ea93fff39b07b995f0290ee47b49 Jul 06 16:31:22.065530 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.065495 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:22.090708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.090683 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5"] Jul 06 16:31:22.091387 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.091362 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:22.111186 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.111158 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xl28k\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:22.132068 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.132043 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6j7pk\" (UniqueName: \"kubernetes.io/projected/ce1c4335-f099-40ec-ae5c-01e27894b3ed-kube-api-access-6j7pk\") pod \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " Jul 06 16:31:22.132190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.132093 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt6qs\" (UniqueName: \"kubernetes.io/projected/aa698e55-ca60-4956-bf15-e0932e1121dd-kube-api-access-lt6qs\") pod \"aa698e55-ca60-4956-bf15-e0932e1121dd\" (UID: \"aa698e55-ca60-4956-bf15-e0932e1121dd\") " Jul 06 16:31:22.132190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.132129 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce1c4335-f099-40ec-ae5c-01e27894b3ed-extensions-socket-volume\") pod \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\" (UID: \"ce1c4335-f099-40ec-ae5c-01e27894b3ed\") " Jul 06 16:31:22.132666 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.132640 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce1c4335-f099-40ec-ae5c-01e27894b3ed-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ce1c4335-f099-40ec-ae5c-01e27894b3ed" (UID: "ce1c4335-f099-40ec-ae5c-01e27894b3ed"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:31:22.134008 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.133983 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce1c4335-f099-40ec-ae5c-01e27894b3ed-kube-api-access-6j7pk" (OuterVolumeSpecName: "kube-api-access-6j7pk") pod "ce1c4335-f099-40ec-ae5c-01e27894b3ed" (UID: "ce1c4335-f099-40ec-ae5c-01e27894b3ed"). InnerVolumeSpecName "kube-api-access-6j7pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:31:22.134085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.134024 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa698e55-ca60-4956-bf15-e0932e1121dd-kube-api-access-lt6qs" (OuterVolumeSpecName: "kube-api-access-lt6qs") pod "aa698e55-ca60-4956-bf15-e0932e1121dd" (UID: "aa698e55-ca60-4956-bf15-e0932e1121dd"). InnerVolumeSpecName "kube-api-access-lt6qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:31:22.233117 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.233085 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j7pk\" (UniqueName: \"kubernetes.io/projected/ce1c4335-f099-40ec-ae5c-01e27894b3ed-kube-api-access-6j7pk\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:31:22.233117 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.233111 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt6qs\" (UniqueName: \"kubernetes.io/projected/aa698e55-ca60-4956-bf15-e0932e1121dd-kube-api-access-lt6qs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:31:22.233117 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.233120 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ce1c4335-f099-40ec-ae5c-01e27894b3ed-extensions-socket-volume\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:31:22.721680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.721649 2563 generic.go:356] "Generic (PLEG): container finished" podID="aa698e55-ca60-4956-bf15-e0932e1121dd" containerID="5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3" exitCode=0 Jul 06 16:31:22.721898 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.721693 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:22.721898 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.721744 2563 scope.go:117] "RemoveContainer" containerID="5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3" Jul 06 16:31:22.723357 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.723337 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" event={"ID":"12c6b817-4094-4629-b554-3ef3b9f714cc","Type":"ContainerStarted","Data":"cc4f64b004582f3d92caac2a94e34b6e78703b50ed84b88068f6fdec7bd0eda4"} Jul 06 16:31:22.724842 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.724827 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:22.727089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.726836 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" event={"ID":"9d418a4f-b754-4ad3-bae1-78eafcc5555f","Type":"ContainerStarted","Data":"8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b"} Jul 06 16:31:22.727089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.726892 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" event={"ID":"9d418a4f-b754-4ad3-bae1-78eafcc5555f","Type":"ContainerStarted","Data":"4775569dcd949e9c70649a1fde5a7dbe3d16ea93fff39b07b995f0290ee47b49"} Jul 06 16:31:22.727089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.726951 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:22.728565 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.728547 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" event={"ID":"91c55469-a3bd-4def-9aaa-7d505c6cf825","Type":"ContainerStarted","Data":"a8eecc9c4f061f779121c9330feea029848f471f54f78e0a0feb110f537dc2dd"} Jul 06 16:31:22.728660 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.728589 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" event={"ID":"91c55469-a3bd-4def-9aaa-7d505c6cf825","Type":"ContainerStarted","Data":"e0778c46a52f4f318d8dbab8d0318c37ebda48aec354a84b2a6c18a8764e6a16"} Jul 06 16:31:22.728715 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.728693 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:22.731233 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.731181 2563 scope.go:117] "RemoveContainer" containerID="5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3" Jul 06 16:31:22.731595 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.731570 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3\": container with ID starting with 5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3 not found: ID does not exist" containerID="5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3" Jul 06 16:31:22.731697 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.731606 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3"} err="failed to get container status \"5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3\": rpc error: code = NotFound desc = could not find container \"5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3\": container with ID starting with 5fd3ed6dd0af9904e08e6d69bc421a6f6782b0d034effa397cc53ce66c82a4b3 not found: ID does not exist" Jul 06 16:31:22.731697 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.731626 2563 scope.go:117] "RemoveContainer" containerID="58ca68b80d69875bbdb6e89646dea387577c170a4e25d4c4f4ba4183f150b913" Jul 06 16:31:22.740431 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.740406 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:22.758595 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.758569 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xl28k\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:22.777352 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.777325 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-xl28k\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-xl28k" Jul 06 16:31:22.875776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.875720 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" podStartSLOduration=15.875705623 podStartE2EDuration="15.875705623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:31:22.831184789 +0000 UTC m=+465.546403774" watchObservedRunningTime="2026-07-06 16:31:22.875705623 +0000 UTC m=+465.590924597" Jul 06 16:31:22.876130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.876098 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" podStartSLOduration=15.876089345 podStartE2EDuration="15.876089345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:31:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:31:22.874353096 +0000 UTC m=+465.589572089" watchObservedRunningTime="2026-07-06 16:31:22.876089345 +0000 UTC m=+465.591308321" Jul 06 16:31:22.882255 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:22.882227 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-2lxzt\" is forbidden: User \"system:node:ip-10-0-136-202.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-202.ec2.internal' and this object" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-2lxzt" Jul 06 16:31:22.913282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:22.913229 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-9hp2k" podStartSLOduration=2.156890504 podStartE2EDuration="26.913212321s" totalImagesPullingTime="24.756321817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:31:22.911771247 +0000 UTC m=+465.626990221" watchObservedRunningTime="2026-07-06 16:31:22.913212321 +0000 UTC m=+465.628431294" Jul 06 16:31:23.796302 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:23.796272 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa698e55-ca60-4956-bf15-e0932e1121dd" path="/var/lib/kubelet/pods/aa698e55-ca60-4956-bf15-e0932e1121dd/volumes" Jul 06 16:31:23.796661 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:23.796649 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce1c4335-f099-40ec-ae5c-01e27894b3ed" path="/var/lib/kubelet/pods/ce1c4335-f099-40ec-ae5c-01e27894b3ed/volumes" Jul 06 16:31:28.299673 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.299637 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5"] Jul 06 16:31:28.300130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.299837 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" podUID="9d418a4f-b754-4ad3-bae1-78eafcc5555f" containerName="manager" containerID="cri-o://8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b" gracePeriod=10 Jul 06 16:31:28.301723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.301700 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:28.544447 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.544424 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:28.688849 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.688814 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d418a4f-b754-4ad3-bae1-78eafcc5555f-extensions-socket-volume\") pod \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " Jul 06 16:31:28.689041 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.688858 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p5m5v\" (UniqueName: \"kubernetes.io/projected/9d418a4f-b754-4ad3-bae1-78eafcc5555f-kube-api-access-p5m5v\") pod \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\" (UID: \"9d418a4f-b754-4ad3-bae1-78eafcc5555f\") " Jul 06 16:31:28.689195 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.689169 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d418a4f-b754-4ad3-bae1-78eafcc5555f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9d418a4f-b754-4ad3-bae1-78eafcc5555f" (UID: "9d418a4f-b754-4ad3-bae1-78eafcc5555f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:31:28.690958 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.690936 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d418a4f-b754-4ad3-bae1-78eafcc5555f-kube-api-access-p5m5v" (OuterVolumeSpecName: "kube-api-access-p5m5v") pod "9d418a4f-b754-4ad3-bae1-78eafcc5555f" (UID: "9d418a4f-b754-4ad3-bae1-78eafcc5555f"). InnerVolumeSpecName "kube-api-access-p5m5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:31:28.760515 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.760481 2563 generic.go:356] "Generic (PLEG): container finished" podID="9d418a4f-b754-4ad3-bae1-78eafcc5555f" containerID="8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b" exitCode=0 Jul 06 16:31:28.760698 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.760525 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" Jul 06 16:31:28.760698 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.760539 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" event={"ID":"9d418a4f-b754-4ad3-bae1-78eafcc5555f","Type":"ContainerDied","Data":"8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b"} Jul 06 16:31:28.760698 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.760590 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5" event={"ID":"9d418a4f-b754-4ad3-bae1-78eafcc5555f","Type":"ContainerDied","Data":"4775569dcd949e9c70649a1fde5a7dbe3d16ea93fff39b07b995f0290ee47b49"} Jul 06 16:31:28.760698 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.760620 2563 scope.go:117] "RemoveContainer" containerID="8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b" Jul 06 16:31:28.769706 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.769684 2563 scope.go:117] "RemoveContainer" containerID="8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b" Jul 06 16:31:28.769989 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:28.769968 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b\": container with ID starting with 8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b not found: ID does not exist" containerID="8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b" Jul 06 16:31:28.770047 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.769998 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b"} err="failed to get container status \"8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b\": rpc error: code = NotFound desc = could not find container \"8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b\": container with ID starting with 8c5630aead59a85a7999d7c6950775a3f199c79ac8958d7a2becc5a2f3a5b69b not found: ID does not exist" Jul 06 16:31:28.789597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.789573 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9d418a4f-b754-4ad3-bae1-78eafcc5555f-extensions-socket-volume\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:31:28.789597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.789595 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5m5v\" (UniqueName: \"kubernetes.io/projected/9d418a4f-b754-4ad3-bae1-78eafcc5555f-kube-api-access-p5m5v\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:31:28.854465 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.854429 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5"] Jul 06 16:31:28.881576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:28.881545 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-hfmg5"] Jul 06 16:31:29.793614 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:29.793582 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d418a4f-b754-4ad3-bae1-78eafcc5555f" path="/var/lib/kubelet/pods/9d418a4f-b754-4ad3-bae1-78eafcc5555f/volumes" Jul 06 16:31:33.737507 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:33.737477 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-tvrm8" Jul 06 16:31:45.154430 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.154397 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln"] Jul 06 16:31:45.154887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.154708 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d418a4f-b754-4ad3-bae1-78eafcc5555f" containerName="manager" Jul 06 16:31:45.154887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.154717 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d418a4f-b754-4ad3-bae1-78eafcc5555f" containerName="manager" Jul 06 16:31:45.154887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.154783 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d418a4f-b754-4ad3-bae1-78eafcc5555f" containerName="manager" Jul 06 16:31:45.338088 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.338052 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln"] Jul 06 16:31:45.338276 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.338180 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.341916 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.341892 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6sqhj\"" Jul 06 16:31:45.428636 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428563 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446c722-3560-485f-9503-7be061763145-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428636 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428597 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446c722-3560-485f-9503-7be061763145-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428636 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428616 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428679 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428707 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446c722-3560-485f-9503-7be061763145-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428832 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.428860 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.428855 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqhz7\" (UniqueName: \"kubernetes.io/projected/d446c722-3560-485f-9503-7be061763145-kube-api-access-cqhz7\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.529934 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.529900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.529934 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.529942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.529974 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.529997 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqhz7\" (UniqueName: \"kubernetes.io/projected/d446c722-3560-485f-9503-7be061763145-kube-api-access-cqhz7\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446c722-3560-485f-9503-7be061763145-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530069 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446c722-3560-485f-9503-7be061763145-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530091 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530429 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446c722-3560-485f-9503-7be061763145-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530429 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530306 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-workload-certs\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530528 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530489 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-credential-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530620 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530595 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-workload-socket\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530751 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530691 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-istio-data\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.530879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.530851 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d446c722-3560-485f-9503-7be061763145-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.532257 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.532240 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d446c722-3560-485f-9503-7be061763145-istio-envoy\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.532465 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.532448 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d446c722-3560-485f-9503-7be061763145-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.541476 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.541445 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d446c722-3560-485f-9503-7be061763145-istio-token\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.541784 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.541767 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqhz7\" (UniqueName: \"kubernetes.io/projected/d446c722-3560-485f-9503-7be061763145-kube-api-access-cqhz7\") pod \"maas-default-gateway-openshift-default-5896d4dd49-4ndln\" (UID: \"d446c722-3560-485f-9503-7be061763145\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.649106 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.649067 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:45.835542 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.835522 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln"] Jul 06 16:31:45.837772 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:31:45.837748 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd446c722_3560_485f_9503_7be061763145.slice/crio-9e9f5e5660441a0b639a7289a49cf653fcce643f1ec58d6555092bfdac0bbc10 WatchSource:0}: Error finding container 9e9f5e5660441a0b639a7289a49cf653fcce643f1ec58d6555092bfdac0bbc10: Status 404 returned error can't find the container with id 9e9f5e5660441a0b639a7289a49cf653fcce643f1ec58d6555092bfdac0bbc10 Jul 06 16:31:45.839847 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.839811 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:31:45.839925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.839879 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:31:45.839969 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:45.839924 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:31:46.844120 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:46.844081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" event={"ID":"d446c722-3560-485f-9503-7be061763145","Type":"ContainerStarted","Data":"67a076feffd50cd0ea0a352fc9a4aed2bc34fd0ade8230b5cb43727d4910dcee"} Jul 06 16:31:46.844539 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:46.844136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" event={"ID":"d446c722-3560-485f-9503-7be061763145","Type":"ContainerStarted","Data":"9e9f5e5660441a0b639a7289a49cf653fcce643f1ec58d6555092bfdac0bbc10"} Jul 06 16:31:46.872373 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:46.872317 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" podStartSLOduration=1.872298292 podStartE2EDuration="1.872298292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:31:46.870411359 +0000 UTC m=+489.585630334" watchObservedRunningTime="2026-07-06 16:31:46.872298292 +0000 UTC m=+489.587517267" Jul 06 16:31:47.649446 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:47.649405 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:47.654248 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:47.654225 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:47.848194 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:47.848163 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:47.849055 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:47.849038 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:31:47.926962 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:47.926889 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:31:48.040530 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.040503 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:31:48.040530 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.040536 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:31:48.040758 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.040620 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.051130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.051111 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 06 16:31:48.154157 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.154120 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z99wd\" (UniqueName: \"kubernetes.io/projected/31016d55-c2db-4251-9670-21a7688142b6-kube-api-access-z99wd\") pod \"limitador-limitador-64c94fd64f-dbzwb\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.154335 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.154180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31016d55-c2db-4251-9670-21a7688142b6-config-file\") pod \"limitador-limitador-64c94fd64f-dbzwb\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.255371 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.255288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z99wd\" (UniqueName: \"kubernetes.io/projected/31016d55-c2db-4251-9670-21a7688142b6-kube-api-access-z99wd\") pod \"limitador-limitador-64c94fd64f-dbzwb\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.255371 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.255350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31016d55-c2db-4251-9670-21a7688142b6-config-file\") pod \"limitador-limitador-64c94fd64f-dbzwb\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.256103 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.256079 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31016d55-c2db-4251-9670-21a7688142b6-config-file\") pod \"limitador-limitador-64c94fd64f-dbzwb\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.267601 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.267575 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z99wd\" (UniqueName: \"kubernetes.io/projected/31016d55-c2db-4251-9670-21a7688142b6-kube-api-access-z99wd\") pod \"limitador-limitador-64c94fd64f-dbzwb\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.351554 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.351515 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:48.522726 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:31:48.522694 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31016d55_c2db_4251_9670_21a7688142b6.slice/crio-68ec6657bdb1162abdf63186977bcc75112a4d3927f7aeb90026ecaed6c1587f WatchSource:0}: Error finding container 68ec6657bdb1162abdf63186977bcc75112a4d3927f7aeb90026ecaed6c1587f: Status 404 returned error can't find the container with id 68ec6657bdb1162abdf63186977bcc75112a4d3927f7aeb90026ecaed6c1587f Jul 06 16:31:48.525784 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.525764 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:31:48.830565 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:48.830490 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-xz9rq"] Jul 06 16:31:49.049493 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.049452 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xz9rq"] Jul 06 16:31:49.049880 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.049496 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" event={"ID":"31016d55-c2db-4251-9670-21a7688142b6","Type":"ContainerStarted","Data":"68ec6657bdb1162abdf63186977bcc75112a4d3927f7aeb90026ecaed6c1587f"} Jul 06 16:31:49.049880 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.049576 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:49.057803 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.057775 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7vfs8\"" Jul 06 16:31:49.162617 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.162587 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btfch\" (UniqueName: \"kubernetes.io/projected/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0-kube-api-access-btfch\") pod \"authorino-58cd955675-xz9rq\" (UID: \"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0\") " pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:49.264212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.264111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btfch\" (UniqueName: \"kubernetes.io/projected/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0-kube-api-access-btfch\") pod \"authorino-58cd955675-xz9rq\" (UID: \"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0\") " pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:49.307576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.307543 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btfch\" (UniqueName: \"kubernetes.io/projected/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0-kube-api-access-btfch\") pod \"authorino-58cd955675-xz9rq\" (UID: \"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0\") " pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:49.358884 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.358858 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:49.530685 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.530584 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-xz9rq"] Jul 06 16:31:49.531138 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:31:49.531109 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b4e5cf7_eac7_4ab5_8a1e_856a74196eb0.slice/crio-3cd6adeb835b4fa13d540fb33fe5d0f96fa9ff6c311143c1d84984e9c156a16a WatchSource:0}: Error finding container 3cd6adeb835b4fa13d540fb33fe5d0f96fa9ff6c311143c1d84984e9c156a16a: Status 404 returned error can't find the container with id 3cd6adeb835b4fa13d540fb33fe5d0f96fa9ff6c311143c1d84984e9c156a16a Jul 06 16:31:49.862268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:49.862201 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xz9rq" event={"ID":"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0","Type":"ContainerStarted","Data":"3cd6adeb835b4fa13d540fb33fe5d0f96fa9ff6c311143c1d84984e9c156a16a"} Jul 06 16:31:53.892282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:53.892184 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" event={"ID":"31016d55-c2db-4251-9670-21a7688142b6","Type":"ContainerStarted","Data":"68185a0ba1b487ab1f6f3768c6b5b8f89379fc980c197dd3d59d7d10ced8dc90"} Jul 06 16:31:53.892282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:53.892255 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:31:53.893701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:53.893679 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xz9rq" event={"ID":"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0","Type":"ContainerStarted","Data":"7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d"} Jul 06 16:31:53.920481 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:53.920429 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" podStartSLOduration=1.807184873 podStartE2EDuration="6.920414301s" totalImagesPullingTime="5.113229428s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:31:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:31:53.918535064 +0000 UTC m=+496.633754056" watchObservedRunningTime="2026-07-06 16:31:53.920414301 +0000 UTC m=+496.635633275" Jul 06 16:31:53.941731 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:53.941674 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-xz9rq" podStartSLOduration=1.788173461 podStartE2EDuration="5.9416561s" totalImagesPullingTime="4.153482639s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:31:53.941130531 +0000 UTC m=+496.656349518" watchObservedRunningTime="2026-07-06 16:31:53.9416561 +0000 UTC m=+496.656875073" Jul 06 16:31:54.451438 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:54.451405 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xz9rq"] Jul 06 16:31:55.903721 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:55.903676 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-xz9rq" podUID="7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" containerName="authorino" containerID="cri-o://7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d" gracePeriod=30 Jul 06 16:31:56.158453 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.158397 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:56.230667 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.230635 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-btfch\" (UniqueName: \"kubernetes.io/projected/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0-kube-api-access-btfch\") pod \"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0\" (UID: \"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0\") " Jul 06 16:31:56.232664 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.232640 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0-kube-api-access-btfch" (OuterVolumeSpecName: "kube-api-access-btfch") pod "7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" (UID: "7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0"). InnerVolumeSpecName "kube-api-access-btfch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:31:56.331909 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.331874 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btfch\" (UniqueName: \"kubernetes.io/projected/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0-kube-api-access-btfch\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:31:56.909560 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.909527 2563 generic.go:356] "Generic (PLEG): container finished" podID="7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" containerID="7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d" exitCode=0 Jul 06 16:31:56.910006 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.909572 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-xz9rq" Jul 06 16:31:56.910006 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.909618 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xz9rq" event={"ID":"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0","Type":"ContainerDied","Data":"7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d"} Jul 06 16:31:56.910006 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.909656 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-xz9rq" event={"ID":"7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0","Type":"ContainerDied","Data":"3cd6adeb835b4fa13d540fb33fe5d0f96fa9ff6c311143c1d84984e9c156a16a"} Jul 06 16:31:56.910006 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.909671 2563 scope.go:117] "RemoveContainer" containerID="7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d" Jul 06 16:31:56.918308 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.918294 2563 scope.go:117] "RemoveContainer" containerID="7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d" Jul 06 16:31:56.918565 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:31:56.918547 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d\": container with ID starting with 7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d not found: ID does not exist" containerID="7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d" Jul 06 16:31:56.918603 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.918573 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d"} err="failed to get container status \"7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d\": rpc error: code = NotFound desc = could not find container \"7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d\": container with ID starting with 7065827b02e984241f635b950dd4dc0b986d9696cda5eaca8b216597963d201d not found: ID does not exist" Jul 06 16:31:56.935899 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.935876 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-xz9rq"] Jul 06 16:31:56.938101 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:56.938081 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-xz9rq"] Jul 06 16:31:57.792971 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:31:57.792941 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" path="/var/lib/kubelet/pods/7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0/volumes" Jul 06 16:32:03.465986 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:03.465946 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:32:03.466392 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:03.466163 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" podUID="31016d55-c2db-4251-9670-21a7688142b6" containerName="limitador" containerID="cri-o://68185a0ba1b487ab1f6f3768c6b5b8f89379fc980c197dd3d59d7d10ced8dc90" gracePeriod=30 Jul 06 16:32:03.466892 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:03.466875 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:32:03.942921 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:03.942889 2563 generic.go:356] "Generic (PLEG): container finished" podID="31016d55-c2db-4251-9670-21a7688142b6" containerID="68185a0ba1b487ab1f6f3768c6b5b8f89379fc980c197dd3d59d7d10ced8dc90" exitCode=0 Jul 06 16:32:03.943100 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:03.942970 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" event={"ID":"31016d55-c2db-4251-9670-21a7688142b6","Type":"ContainerDied","Data":"68185a0ba1b487ab1f6f3768c6b5b8f89379fc980c197dd3d59d7d10ced8dc90"} Jul 06 16:32:04.014212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.014183 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:32:04.100952 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.100922 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z99wd\" (UniqueName: \"kubernetes.io/projected/31016d55-c2db-4251-9670-21a7688142b6-kube-api-access-z99wd\") pod \"31016d55-c2db-4251-9670-21a7688142b6\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " Jul 06 16:32:04.101172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.100960 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31016d55-c2db-4251-9670-21a7688142b6-config-file\") pod \"31016d55-c2db-4251-9670-21a7688142b6\" (UID: \"31016d55-c2db-4251-9670-21a7688142b6\") " Jul 06 16:32:04.103521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.101582 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31016d55-c2db-4251-9670-21a7688142b6-config-file" (OuterVolumeSpecName: "config-file") pod "31016d55-c2db-4251-9670-21a7688142b6" (UID: "31016d55-c2db-4251-9670-21a7688142b6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:32:04.105857 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.104060 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31016d55-c2db-4251-9670-21a7688142b6-kube-api-access-z99wd" (OuterVolumeSpecName: "kube-api-access-z99wd") pod "31016d55-c2db-4251-9670-21a7688142b6" (UID: "31016d55-c2db-4251-9670-21a7688142b6"). InnerVolumeSpecName "kube-api-access-z99wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:32:04.202483 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.202444 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z99wd\" (UniqueName: \"kubernetes.io/projected/31016d55-c2db-4251-9670-21a7688142b6-kube-api-access-z99wd\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:32:04.202483 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.202479 2563 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/31016d55-c2db-4251-9670-21a7688142b6-config-file\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:32:04.948828 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.948802 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" Jul 06 16:32:04.948828 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.948786 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-dbzwb" event={"ID":"31016d55-c2db-4251-9670-21a7688142b6","Type":"ContainerDied","Data":"68ec6657bdb1162abdf63186977bcc75112a4d3927f7aeb90026ecaed6c1587f"} Jul 06 16:32:04.949308 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:04.948872 2563 scope.go:117] "RemoveContainer" containerID="68185a0ba1b487ab1f6f3768c6b5b8f89379fc980c197dd3d59d7d10ced8dc90" Jul 06 16:32:05.033549 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:05.033508 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:32:05.052399 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:05.052371 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-dbzwb"] Jul 06 16:32:05.792879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:05.792840 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31016d55-c2db-4251-9670-21a7688142b6" path="/var/lib/kubelet/pods/31016d55-c2db-4251-9670-21a7688142b6/volumes" Jul 06 16:32:10.627510 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627480 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-wmhxp"] Jul 06 16:32:10.627918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627822 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" containerName="authorino" Jul 06 16:32:10.627918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627834 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" containerName="authorino" Jul 06 16:32:10.627918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627850 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31016d55-c2db-4251-9670-21a7688142b6" containerName="limitador" Jul 06 16:32:10.627918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627855 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="31016d55-c2db-4251-9670-21a7688142b6" containerName="limitador" Jul 06 16:32:10.627918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627918 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="31016d55-c2db-4251-9670-21a7688142b6" containerName="limitador" Jul 06 16:32:10.628076 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.627928 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b4e5cf7-eac7-4ab5-8a1e-856a74196eb0" containerName="authorino" Jul 06 16:32:10.630688 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.630673 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.649235 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.649208 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 06 16:32:10.649983 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.649961 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-wmhxp"] Jul 06 16:32:10.665015 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.664988 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-p2q5f\"" Jul 06 16:32:10.760046 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.760008 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh2vb\" (UniqueName: \"kubernetes.io/projected/076b7c9d-4a18-4de3-aa35-ebd1680a0729-kube-api-access-sh2vb\") pod \"postgres-69fb4594cf-wmhxp\" (UID: \"076b7c9d-4a18-4de3-aa35-ebd1680a0729\") " pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.760219 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.760085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/076b7c9d-4a18-4de3-aa35-ebd1680a0729-data\") pod \"postgres-69fb4594cf-wmhxp\" (UID: \"076b7c9d-4a18-4de3-aa35-ebd1680a0729\") " pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.860510 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.860470 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/076b7c9d-4a18-4de3-aa35-ebd1680a0729-data\") pod \"postgres-69fb4594cf-wmhxp\" (UID: \"076b7c9d-4a18-4de3-aa35-ebd1680a0729\") " pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.860682 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.860541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh2vb\" (UniqueName: \"kubernetes.io/projected/076b7c9d-4a18-4de3-aa35-ebd1680a0729-kube-api-access-sh2vb\") pod \"postgres-69fb4594cf-wmhxp\" (UID: \"076b7c9d-4a18-4de3-aa35-ebd1680a0729\") " pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.860884 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.860864 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/076b7c9d-4a18-4de3-aa35-ebd1680a0729-data\") pod \"postgres-69fb4594cf-wmhxp\" (UID: \"076b7c9d-4a18-4de3-aa35-ebd1680a0729\") " pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.885704 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.885624 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh2vb\" (UniqueName: \"kubernetes.io/projected/076b7c9d-4a18-4de3-aa35-ebd1680a0729-kube-api-access-sh2vb\") pod \"postgres-69fb4594cf-wmhxp\" (UID: \"076b7c9d-4a18-4de3-aa35-ebd1680a0729\") " pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:10.947508 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:10.947471 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:11.096836 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:32:11.096806 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076b7c9d_4a18_4de3_aa35_ebd1680a0729.slice/crio-2d8b06a387ee1fbf316ce4f28e4f7d04f4619bf6b5f1b8d7731388632a7bce6e WatchSource:0}: Error finding container 2d8b06a387ee1fbf316ce4f28e4f7d04f4619bf6b5f1b8d7731388632a7bce6e: Status 404 returned error can't find the container with id 2d8b06a387ee1fbf316ce4f28e4f7d04f4619bf6b5f1b8d7731388632a7bce6e Jul 06 16:32:11.101155 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:11.101129 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-wmhxp"] Jul 06 16:32:11.979297 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:11.979259 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-wmhxp" event={"ID":"076b7c9d-4a18-4de3-aa35-ebd1680a0729","Type":"ContainerStarted","Data":"2d8b06a387ee1fbf316ce4f28e4f7d04f4619bf6b5f1b8d7731388632a7bce6e"} Jul 06 16:32:17.009884 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:17.009847 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-wmhxp" event={"ID":"076b7c9d-4a18-4de3-aa35-ebd1680a0729","Type":"ContainerStarted","Data":"b110b6e0e0e8d869c4d2a3540a29be1b3d646629865b1fd270be533cb8e54cef"} Jul 06 16:32:17.010451 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:17.010038 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:17.102931 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:17.102875 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-wmhxp" podStartSLOduration=1.784100215 podStartE2EDuration="7.102860487s" totalImagesPullingTime="5.318760272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:32:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:32:17.102518953 +0000 UTC m=+519.817737937" watchObservedRunningTime="2026-07-06 16:32:17.102860487 +0000 UTC m=+519.818079460" Jul 06 16:32:23.044048 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:23.044022 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-wmhxp" Jul 06 16:32:31.520810 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.516452 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-548dd868df-z6dzn"] Jul 06 16:32:31.521719 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.521694 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:31.537732 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.537706 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 16:32:31.540637 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.540603 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-548dd868df-z6dzn"] Jul 06 16:32:31.560022 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.559998 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 16:32:31.560022 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.560000 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ckncn\"" Jul 06 16:32:31.632680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.632648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:31.632859 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.632724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cp8b\" (UniqueName: \"kubernetes.io/projected/cd9a0b72-955e-433a-98d8-1573b931b317-kube-api-access-2cp8b\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:31.733409 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.733374 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cp8b\" (UniqueName: \"kubernetes.io/projected/cd9a0b72-955e-433a-98d8-1573b931b317-kube-api-access-2cp8b\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:31.733574 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.733467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:31.733574 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:32:31.733565 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 16:32:31.733651 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:32:31.733631 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert podName:cd9a0b72-955e-433a-98d8-1573b931b317 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:32.233615208 +0000 UTC m=+534.948834162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert") pod "maas-controller-548dd868df-z6dzn" (UID: "cd9a0b72-955e-433a-98d8-1573b931b317") : secret "maas-controller-webhook-cert" not found Jul 06 16:32:31.742593 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.742569 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66fcc46d69-m4lh8"] Jul 06 16:32:31.744904 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.744889 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:31.754291 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.754271 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cp8b\" (UniqueName: \"kubernetes.io/projected/cd9a0b72-955e-433a-98d8-1573b931b317-kube-api-access-2cp8b\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:31.770551 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.770530 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-m4lh8"] Jul 06 16:32:31.834461 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.834385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kttld\" (UniqueName: \"kubernetes.io/projected/2d5edfb5-6858-40c6-8e67-19a001299400-kube-api-access-kttld\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:31.834461 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.834416 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:31.935725 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.935694 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:31.935914 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.935846 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kttld\" (UniqueName: \"kubernetes.io/projected/2d5edfb5-6858-40c6-8e67-19a001299400-kube-api-access-kttld\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:31.935914 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:32:31.935888 2563 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 06 16:32:31.935998 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:32:31.935950 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert podName:2d5edfb5-6858-40c6-8e67-19a001299400 nodeName:}" failed. No retries permitted until 2026-07-06 16:32:32.435934212 +0000 UTC m=+535.151153170 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert") pod "maas-controller-66fcc46d69-m4lh8" (UID: "2d5edfb5-6858-40c6-8e67-19a001299400") : secret "maas-controller-webhook-cert" not found Jul 06 16:32:31.956767 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:31.956741 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kttld\" (UniqueName: \"kubernetes.io/projected/2d5edfb5-6858-40c6-8e67-19a001299400-kube-api-access-kttld\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:32.237747 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.237707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:32.240293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.240259 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert\") pod \"maas-controller-548dd868df-z6dzn\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:32.433861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.433830 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:32.440116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.440097 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:32.442341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.442319 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert\") pod \"maas-controller-66fcc46d69-m4lh8\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:32.570385 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:32:32.570352 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9a0b72_955e_433a_98d8_1573b931b317.slice/crio-5620e53e4df01f671461a25d6defecce95bbe163c4ce36d7522e9cec3c3222be WatchSource:0}: Error finding container 5620e53e4df01f671461a25d6defecce95bbe163c4ce36d7522e9cec3c3222be: Status 404 returned error can't find the container with id 5620e53e4df01f671461a25d6defecce95bbe163c4ce36d7522e9cec3c3222be Jul 06 16:32:32.605944 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.605913 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-548dd868df-z6dzn"] Jul 06 16:32:32.654744 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.654717 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:32.793764 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:32:32.793741 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5edfb5_6858_40c6_8e67_19a001299400.slice/crio-a3083437d137adee52a59b08d97e8ca3338432c0c9301ab775fd8082c9ba5088 WatchSource:0}: Error finding container a3083437d137adee52a59b08d97e8ca3338432c0c9301ab775fd8082c9ba5088: Status 404 returned error can't find the container with id a3083437d137adee52a59b08d97e8ca3338432c0c9301ab775fd8082c9ba5088 Jul 06 16:32:32.800971 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:32.800946 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-m4lh8"] Jul 06 16:32:33.088816 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:33.088767 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" event={"ID":"2d5edfb5-6858-40c6-8e67-19a001299400","Type":"ContainerStarted","Data":"a3083437d137adee52a59b08d97e8ca3338432c0c9301ab775fd8082c9ba5088"} Jul 06 16:32:33.089824 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:33.089787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-548dd868df-z6dzn" event={"ID":"cd9a0b72-955e-433a-98d8-1573b931b317","Type":"ContainerStarted","Data":"5620e53e4df01f671461a25d6defecce95bbe163c4ce36d7522e9cec3c3222be"} Jul 06 16:32:35.106165 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:35.106124 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-548dd868df-z6dzn" event={"ID":"cd9a0b72-955e-433a-98d8-1573b931b317","Type":"ContainerStarted","Data":"22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8"} Jul 06 16:32:35.106593 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:35.106181 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:35.107631 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:35.107610 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" event={"ID":"2d5edfb5-6858-40c6-8e67-19a001299400","Type":"ContainerStarted","Data":"1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d"} Jul 06 16:32:35.107765 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:35.107722 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:35.510542 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:35.510415 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" podStartSLOduration=2.55987441 podStartE2EDuration="4.510398686s" totalImagesPullingTime="1.950524276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:32:35.509295999 +0000 UTC m=+538.224514975" watchObservedRunningTime="2026-07-06 16:32:35.510398686 +0000 UTC m=+538.225617659" Jul 06 16:32:35.510723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:35.510688 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-548dd868df-z6dzn" podStartSLOduration=2.345503341 podStartE2EDuration="4.510678592s" totalImagesPullingTime="2.165175251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:32:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:32:35.258616967 +0000 UTC m=+537.973835960" watchObservedRunningTime="2026-07-06 16:32:35.510678592 +0000 UTC m=+538.225897565" Jul 06 16:32:46.121457 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.121428 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:32:46.121980 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.121504 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:46.236691 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.236658 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-548dd868df-z6dzn"] Jul 06 16:32:46.236865 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.236843 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-548dd868df-z6dzn" podUID="cd9a0b72-955e-433a-98d8-1573b931b317" containerName="manager" containerID="cri-o://22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8" gracePeriod=30 Jul 06 16:32:46.481307 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.481282 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:46.566090 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.566054 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2cp8b\" (UniqueName: \"kubernetes.io/projected/cd9a0b72-955e-433a-98d8-1573b931b317-kube-api-access-2cp8b\") pod \"cd9a0b72-955e-433a-98d8-1573b931b317\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " Jul 06 16:32:46.566269 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.566156 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert\") pod \"cd9a0b72-955e-433a-98d8-1573b931b317\" (UID: \"cd9a0b72-955e-433a-98d8-1573b931b317\") " Jul 06 16:32:46.568281 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.568252 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9a0b72-955e-433a-98d8-1573b931b317-kube-api-access-2cp8b" (OuterVolumeSpecName: "kube-api-access-2cp8b") pod "cd9a0b72-955e-433a-98d8-1573b931b317" (UID: "cd9a0b72-955e-433a-98d8-1573b931b317"). InnerVolumeSpecName "kube-api-access-2cp8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:32:46.568394 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.568327 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "cd9a0b72-955e-433a-98d8-1573b931b317" (UID: "cd9a0b72-955e-433a-98d8-1573b931b317"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:32:46.667762 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.667683 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cd9a0b72-955e-433a-98d8-1573b931b317-webhook-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:32:46.667762 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:46.667711 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cp8b\" (UniqueName: \"kubernetes.io/projected/cd9a0b72-955e-433a-98d8-1573b931b317-kube-api-access-2cp8b\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:32:47.165259 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.165228 2563 generic.go:356] "Generic (PLEG): container finished" podID="cd9a0b72-955e-433a-98d8-1573b931b317" containerID="22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8" exitCode=0 Jul 06 16:32:47.165712 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.165269 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-548dd868df-z6dzn" Jul 06 16:32:47.165712 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.165279 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-548dd868df-z6dzn" event={"ID":"cd9a0b72-955e-433a-98d8-1573b931b317","Type":"ContainerDied","Data":"22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8"} Jul 06 16:32:47.165712 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.165321 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-548dd868df-z6dzn" event={"ID":"cd9a0b72-955e-433a-98d8-1573b931b317","Type":"ContainerDied","Data":"5620e53e4df01f671461a25d6defecce95bbe163c4ce36d7522e9cec3c3222be"} Jul 06 16:32:47.165712 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.165345 2563 scope.go:117] "RemoveContainer" containerID="22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8" Jul 06 16:32:47.174241 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.174223 2563 scope.go:117] "RemoveContainer" containerID="22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8" Jul 06 16:32:47.174473 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:32:47.174456 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8\": container with ID starting with 22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8 not found: ID does not exist" containerID="22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8" Jul 06 16:32:47.174519 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.174481 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8"} err="failed to get container status \"22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8\": rpc error: code = NotFound desc = could not find container \"22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8\": container with ID starting with 22385547d1932215be09c8037886a668812c696f664f13fb46a65663a936cfd8 not found: ID does not exist" Jul 06 16:32:47.268632 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.268596 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-548dd868df-z6dzn"] Jul 06 16:32:47.338303 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.338269 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-548dd868df-z6dzn"] Jul 06 16:32:47.793226 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:32:47.793196 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd9a0b72-955e-433a-98d8-1573b931b317" path="/var/lib/kubelet/pods/cd9a0b72-955e-433a-98d8-1573b931b317/volumes" Jul 06 16:33:07.925522 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.925489 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-dsvxc"] Jul 06 16:33:07.926208 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.926183 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd9a0b72-955e-433a-98d8-1573b931b317" containerName="manager" Jul 06 16:33:07.926208 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.926210 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd9a0b72-955e-433a-98d8-1573b931b317" containerName="manager" Jul 06 16:33:07.926455 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.926302 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd9a0b72-955e-433a-98d8-1573b931b317" containerName="manager" Jul 06 16:33:07.928590 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.928570 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:07.939666 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.939638 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-dsvxc"] Jul 06 16:33:07.946043 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.946018 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c32da6e-f25c-4c71-a97e-db398d6116ef-plugins-config-volume\") pod \"payload-processing-58f86c7798-dsvxc\" (UID: \"9c32da6e-f25c-4c71-a97e-db398d6116ef\") " pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:07.946161 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.946119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwcm2\" (UniqueName: \"kubernetes.io/projected/9c32da6e-f25c-4c71-a97e-db398d6116ef-kube-api-access-rwcm2\") pod \"payload-processing-58f86c7798-dsvxc\" (UID: \"9c32da6e-f25c-4c71-a97e-db398d6116ef\") " pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:07.952988 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.952967 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-s8ljj\"" Jul 06 16:33:07.953110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:07.952968 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 06 16:33:08.046617 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:08.046583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwcm2\" (UniqueName: \"kubernetes.io/projected/9c32da6e-f25c-4c71-a97e-db398d6116ef-kube-api-access-rwcm2\") pod \"payload-processing-58f86c7798-dsvxc\" (UID: \"9c32da6e-f25c-4c71-a97e-db398d6116ef\") " pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:08.046787 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:08.046631 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c32da6e-f25c-4c71-a97e-db398d6116ef-plugins-config-volume\") pod \"payload-processing-58f86c7798-dsvxc\" (UID: \"9c32da6e-f25c-4c71-a97e-db398d6116ef\") " pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:08.047380 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:08.047360 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c32da6e-f25c-4c71-a97e-db398d6116ef-plugins-config-volume\") pod \"payload-processing-58f86c7798-dsvxc\" (UID: \"9c32da6e-f25c-4c71-a97e-db398d6116ef\") " pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:08.075381 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:08.075343 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwcm2\" (UniqueName: \"kubernetes.io/projected/9c32da6e-f25c-4c71-a97e-db398d6116ef-kube-api-access-rwcm2\") pod \"payload-processing-58f86c7798-dsvxc\" (UID: \"9c32da6e-f25c-4c71-a97e-db398d6116ef\") " pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:08.242924 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:08.242827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:08.420355 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:33:08.420322 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c32da6e_f25c_4c71_a97e_db398d6116ef.slice/crio-1088f9ed6f2d94e7a9172d767a68f5e9a72968f6c1a66ee8e11fde515b29b42a WatchSource:0}: Error finding container 1088f9ed6f2d94e7a9172d767a68f5e9a72968f6c1a66ee8e11fde515b29b42a: Status 404 returned error can't find the container with id 1088f9ed6f2d94e7a9172d767a68f5e9a72968f6c1a66ee8e11fde515b29b42a Jul 06 16:33:08.422494 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:08.422467 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-dsvxc"] Jul 06 16:33:09.267845 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:09.267815 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" event={"ID":"9c32da6e-f25c-4c71-a97e-db398d6116ef","Type":"ContainerStarted","Data":"1088f9ed6f2d94e7a9172d767a68f5e9a72968f6c1a66ee8e11fde515b29b42a"} Jul 06 16:33:11.279671 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:11.279631 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" event={"ID":"9c32da6e-f25c-4c71-a97e-db398d6116ef","Type":"ContainerStarted","Data":"2b5df98cc88e09ecf331b1536215e6b3a37a23ea0f5a5ab6df89fe3118b15ccb"} Jul 06 16:33:11.280112 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:11.279766 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:11.376915 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:11.376862 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" podStartSLOduration=2.580771252 podStartE2EDuration="4.37684548s" totalImagesPullingTime="1.796074228s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:11.376003943 +0000 UTC m=+574.091222917" watchObservedRunningTime="2026-07-06 16:33:11.37684548 +0000 UTC m=+574.092064454" Jul 06 16:33:15.300844 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:15.300817 2563 generic.go:356] "Generic (PLEG): container finished" podID="d446c722-3560-485f-9503-7be061763145" containerID="67a076feffd50cd0ea0a352fc9a4aed2bc34fd0ade8230b5cb43727d4910dcee" exitCode=0 Jul 06 16:33:15.301199 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:15.300880 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" event={"ID":"d446c722-3560-485f-9503-7be061763145","Type":"ContainerDied","Data":"67a076feffd50cd0ea0a352fc9a4aed2bc34fd0ade8230b5cb43727d4910dcee"} Jul 06 16:33:15.301311 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:15.301291 2563 scope.go:117] "RemoveContainer" containerID="67a076feffd50cd0ea0a352fc9a4aed2bc34fd0ade8230b5cb43727d4910dcee" Jul 06 16:33:15.302292 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:15.302264 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:33:15.302340 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:15.302331 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:33:15.302373 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:15.302364 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:33:16.311768 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:16.310915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" event={"ID":"d446c722-3560-485f-9503-7be061763145","Type":"ContainerStarted","Data":"9db308243bb0f3e198c74d939ae5b6a7c1c1d68b34f280852f0381afb7525773"} Jul 06 16:33:16.649713 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:16.649673 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:33:16.650636 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:16.650607 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-4ndln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 16:33:16.650762 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:16.650656 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" podUID="d446c722-3560-485f-9503-7be061763145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 16:33:17.649687 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:17.649647 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-4ndln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 16:33:17.650112 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:17.649708 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" podUID="d446c722-3560-485f-9503-7be061763145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 16:33:17.848878 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:17.848832 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:33:18.650278 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:18.650244 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-4ndln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 16:33:18.650745 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:18.650309 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" podUID="d446c722-3560-485f-9503-7be061763145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 16:33:19.650003 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:19.649968 2563 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5896d4dd49-4ndln container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" start-of-body= Jul 06 16:33:19.650187 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:19.650019 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" podUID="d446c722-3560-485f-9503-7be061763145" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.35:15021/healthz/ready\": dial tcp 10.133.0.35:15021: connect: connection refused" Jul 06 16:33:20.654086 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:20.654048 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:33:20.655182 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:20.655164 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5896d4dd49-4ndln" Jul 06 16:33:22.285925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:22.285896 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-dsvxc" Jul 06 16:33:35.045849 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.045814 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d"] Jul 06 16:33:35.049309 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.049289 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.102732 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.102705 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vgdcs\"" Jul 06 16:33:35.104051 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.104023 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d"] Jul 06 16:33:35.127345 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.127323 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 06 16:33:35.148239 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.148207 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 06 16:33:35.170622 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.170600 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 06 16:33:35.191037 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.191010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.191137 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.191076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.191137 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.191129 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6982831b-656b-4046-91a8-26e5eede3b00-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.191225 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.191159 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.191225 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.191190 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.191301 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.191223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxzwn\" (UniqueName: \"kubernetes.io/projected/6982831b-656b-4046-91a8-26e5eede3b00-kube-api-access-cxzwn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292368 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292335 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292368 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292372 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292393 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxzwn\" (UniqueName: \"kubernetes.io/projected/6982831b-656b-4046-91a8-26e5eede3b00-kube-api-access-cxzwn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292447 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292597 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6982831b-656b-4046-91a8-26e5eede3b00-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.292888 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.292856 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.293113 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.293086 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.293342 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.293315 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.294923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.294895 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6982831b-656b-4046-91a8-26e5eede3b00-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.295063 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.295047 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6982831b-656b-4046-91a8-26e5eede3b00-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.331491 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.331425 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxzwn\" (UniqueName: \"kubernetes.io/projected/6982831b-656b-4046-91a8-26e5eede3b00-kube-api-access-cxzwn\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d\" (UID: \"6982831b-656b-4046-91a8-26e5eede3b00\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.359426 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.359391 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:35.549064 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:33:35.549032 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6982831b_656b_4046_91a8_26e5eede3b00.slice/crio-d28abd4dc3247b3d06f2c49236fbd798a4f7d6a73551caa92b95a9f67b4dde4f WatchSource:0}: Error finding container d28abd4dc3247b3d06f2c49236fbd798a4f7d6a73551caa92b95a9f67b4dde4f: Status 404 returned error can't find the container with id d28abd4dc3247b3d06f2c49236fbd798a4f7d6a73551caa92b95a9f67b4dde4f Jul 06 16:33:35.550001 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:35.549979 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d"] Jul 06 16:33:36.409140 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:36.409105 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" event={"ID":"6982831b-656b-4046-91a8-26e5eede3b00","Type":"ContainerStarted","Data":"d28abd4dc3247b3d06f2c49236fbd798a4f7d6a73551caa92b95a9f67b4dde4f"} Jul 06 16:33:37.414941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:37.414906 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" event={"ID":"6982831b-656b-4046-91a8-26e5eede3b00","Type":"ContainerStarted","Data":"55e621a83009ca512f042ba303f582cc37ca0fe01bb74a4a752b7abce2472e63"} Jul 06 16:33:37.731315 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:37.731236 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:33:37.731454 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:37.731388 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:33:45.359986 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:45.359941 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:45.360499 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:45.360018 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:45.372695 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:45.372665 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:33:45.462051 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:45.461998 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" podStartSLOduration=10.091866722 podStartE2EDuration="11.461982657s" totalImagesPullingTime="1.370115935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:33:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:33:37.722282709 +0000 UTC m=+600.437501704" watchObservedRunningTime="2026-07-06 16:33:45.461982657 +0000 UTC m=+608.177201631" Jul 06 16:33:45.466345 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:33:45.466325 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d" Jul 06 16:34:06.533064 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.532949 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw"] Jul 06 16:34:06.536476 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.536455 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.556861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.556830 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.556861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.556865 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.557090 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.556881 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsvx\" (UniqueName: \"kubernetes.io/projected/7441d830-e5de-4cf1-a6db-7b07d0381955-kube-api-access-nnsvx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.557090 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.556981 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7441d830-e5de-4cf1-a6db-7b07d0381955-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.557090 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.557010 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.557190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.557094 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.560252 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.560228 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw"] Jul 06 16:34:06.630414 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.630378 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 06 16:34:06.657727 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.657691 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.657941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.657747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.657941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.657775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.657941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.657827 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsvx\" (UniqueName: \"kubernetes.io/projected/7441d830-e5de-4cf1-a6db-7b07d0381955-kube-api-access-nnsvx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.657941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.657902 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7441d830-e5de-4cf1-a6db-7b07d0381955-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.657941 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.657934 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.658390 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.658198 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.658390 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.658231 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.658390 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.658279 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.660194 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.660174 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7441d830-e5de-4cf1-a6db-7b07d0381955-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.660355 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.660336 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7441d830-e5de-4cf1-a6db-7b07d0381955-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.723102 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.723068 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsvx\" (UniqueName: \"kubernetes.io/projected/7441d830-e5de-4cf1-a6db-7b07d0381955-kube-api-access-nnsvx\") pod \"e2e-trlp-test-simulated-kserve-598466548d-7mtgw\" (UID: \"7441d830-e5de-4cf1-a6db-7b07d0381955\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.846853 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.846827 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:06.992199 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:34:06.992168 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7441d830_e5de_4cf1_a6db_7b07d0381955.slice/crio-90933b32326fdeb19694ef6fbba03d2daa747c44424754bf4de5c4523d1f8817 WatchSource:0}: Error finding container 90933b32326fdeb19694ef6fbba03d2daa747c44424754bf4de5c4523d1f8817: Status 404 returned error can't find the container with id 90933b32326fdeb19694ef6fbba03d2daa747c44424754bf4de5c4523d1f8817 Jul 06 16:34:06.993907 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:06.993887 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:34:07.026888 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:07.026843 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw"] Jul 06 16:34:07.562104 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:07.561993 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" event={"ID":"7441d830-e5de-4cf1-a6db-7b07d0381955","Type":"ContainerStarted","Data":"9918f70ea470a26c509464cfed5071ac21bdddfc309076ec5348f5d5b3357557"} Jul 06 16:34:07.562104 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:07.562060 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" event={"ID":"7441d830-e5de-4cf1-a6db-7b07d0381955","Type":"ContainerStarted","Data":"90933b32326fdeb19694ef6fbba03d2daa747c44424754bf4de5c4523d1f8817"} Jul 06 16:34:07.954420 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:07.954368 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" podStartSLOduration=1.7607748079999999 podStartE2EDuration="1.954350402s" totalImagesPullingTime="193.575594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:34:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:34:07.928373192 +0000 UTC m=+630.643592176" watchObservedRunningTime="2026-07-06 16:34:07.954350402 +0000 UTC m=+630.669569377" Jul 06 16:34:16.848059 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:16.848021 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:16.848471 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:16.848086 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:16.860492 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:16.860467 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:17.622839 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:17.622813 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-7mtgw" Jul 06 16:34:36.510899 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.510868 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj"] Jul 06 16:34:36.515029 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.515001 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj"] Jul 06 16:34:36.515153 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.515109 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.563218 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.563184 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 06 16:34:36.634331 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.634293 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884fh\" (UniqueName: \"kubernetes.io/projected/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-kube-api-access-884fh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.634331 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.634336 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.634576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.634396 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.634576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.634432 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.634576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.634471 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.634576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.634491 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.735722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.735690 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.735924 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.735853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.735924 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.735893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.736025 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.735945 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884fh\" (UniqueName: \"kubernetes.io/projected/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-kube-api-access-884fh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.736025 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.735972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.736025 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.736006 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.736176 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.736037 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.736234 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.736171 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.736413 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.736391 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.738108 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.738084 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.738342 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.738326 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:36.870672 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:36.870635 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-884fh\" (UniqueName: \"kubernetes.io/projected/f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b-kube-api-access-884fh\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-b2nqj\" (UID: \"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:37.125607 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:37.125512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:37.272702 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:34:37.272675 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e0fa44_5e38_4a19_bb80_f8e70ea2c59b.slice/crio-fc61437a1cff8fae87e6cbc22d30564e3f0a5c163d8750f25e0cc5bd594f381e WatchSource:0}: Error finding container fc61437a1cff8fae87e6cbc22d30564e3f0a5c163d8750f25e0cc5bd594f381e: Status 404 returned error can't find the container with id fc61437a1cff8fae87e6cbc22d30564e3f0a5c163d8750f25e0cc5bd594f381e Jul 06 16:34:37.382130 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:37.382047 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj"] Jul 06 16:34:37.714771 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:37.714699 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" event={"ID":"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b","Type":"ContainerStarted","Data":"6f8faeb2bb23ed19fc68c193028d9dadefc9fe7a14bc5f3ac18eb87d120b8634"} Jul 06 16:34:37.714771 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:37.714748 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" event={"ID":"f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b","Type":"ContainerStarted","Data":"fc61437a1cff8fae87e6cbc22d30564e3f0a5c163d8750f25e0cc5bd594f381e"} Jul 06 16:34:37.941032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:37.940983 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" podStartSLOduration=1.7562510310000001 podStartE2EDuration="1.940966516s" totalImagesPullingTime="184.715485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:34:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:34:37.939183715 +0000 UTC m=+660.654402689" watchObservedRunningTime="2026-07-06 16:34:37.940966516 +0000 UTC m=+660.656185489" Jul 06 16:34:47.126682 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:47.126587 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:47.127106 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:47.126736 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:47.138843 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:47.138818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:34:47.771446 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:34:47.771416 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-b2nqj" Jul 06 16:37:07.143813 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.143708 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-m4lh8"] Jul 06 16:37:07.144334 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.143998 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" podUID="2d5edfb5-6858-40c6-8e67-19a001299400" containerName="manager" containerID="cri-o://1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d" gracePeriod=30 Jul 06 16:37:07.403010 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.402943 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:37:07.436128 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.436091 2563 generic.go:356] "Generic (PLEG): container finished" podID="2d5edfb5-6858-40c6-8e67-19a001299400" containerID="1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d" exitCode=0 Jul 06 16:37:07.436300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.436133 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" Jul 06 16:37:07.436300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.436171 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" event={"ID":"2d5edfb5-6858-40c6-8e67-19a001299400","Type":"ContainerDied","Data":"1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d"} Jul 06 16:37:07.436300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.436228 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fcc46d69-m4lh8" event={"ID":"2d5edfb5-6858-40c6-8e67-19a001299400","Type":"ContainerDied","Data":"a3083437d137adee52a59b08d97e8ca3338432c0c9301ab775fd8082c9ba5088"} Jul 06 16:37:07.436300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.436258 2563 scope.go:117] "RemoveContainer" containerID="1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d" Jul 06 16:37:07.447203 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.447182 2563 scope.go:117] "RemoveContainer" containerID="1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d" Jul 06 16:37:07.447560 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:37:07.447537 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d\": container with ID starting with 1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d not found: ID does not exist" containerID="1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d" Jul 06 16:37:07.447618 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.447575 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d"} err="failed to get container status \"1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d\": rpc error: code = NotFound desc = could not find container \"1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d\": container with ID starting with 1e5a340261df269b154617501b02109bd5297149f1769f44d818ac6cda26fc2d not found: ID does not exist" Jul 06 16:37:07.478754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.478721 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kttld\" (UniqueName: \"kubernetes.io/projected/2d5edfb5-6858-40c6-8e67-19a001299400-kube-api-access-kttld\") pod \"2d5edfb5-6858-40c6-8e67-19a001299400\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " Jul 06 16:37:07.478935 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.478828 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert\") pod \"2d5edfb5-6858-40c6-8e67-19a001299400\" (UID: \"2d5edfb5-6858-40c6-8e67-19a001299400\") " Jul 06 16:37:07.480848 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.480812 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "2d5edfb5-6858-40c6-8e67-19a001299400" (UID: "2d5edfb5-6858-40c6-8e67-19a001299400"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 06 16:37:07.480965 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.480938 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5edfb5-6858-40c6-8e67-19a001299400-kube-api-access-kttld" (OuterVolumeSpecName: "kube-api-access-kttld") pod "2d5edfb5-6858-40c6-8e67-19a001299400" (UID: "2d5edfb5-6858-40c6-8e67-19a001299400"). InnerVolumeSpecName "kube-api-access-kttld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:37:07.579556 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.579518 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kttld\" (UniqueName: \"kubernetes.io/projected/2d5edfb5-6858-40c6-8e67-19a001299400-kube-api-access-kttld\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:37:07.579556 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.579550 2563 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2d5edfb5-6858-40c6-8e67-19a001299400-webhook-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:37:07.768352 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.768324 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-m4lh8"] Jul 06 16:37:07.778825 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.778775 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-m4lh8"] Jul 06 16:37:07.793403 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:07.793372 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d5edfb5-6858-40c6-8e67-19a001299400" path="/var/lib/kubelet/pods/2d5edfb5-6858-40c6-8e67-19a001299400/volumes" Jul 06 16:37:08.571754 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.571714 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-66fcc46d69-wprqg"] Jul 06 16:37:08.572311 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.572291 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5edfb5-6858-40c6-8e67-19a001299400" containerName="manager" Jul 06 16:37:08.572403 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.572314 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5edfb5-6858-40c6-8e67-19a001299400" containerName="manager" Jul 06 16:37:08.572457 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.572442 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5edfb5-6858-40c6-8e67-19a001299400" containerName="manager" Jul 06 16:37:08.576785 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.576762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.581178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.581157 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-ckncn\"" Jul 06 16:37:08.581178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.581169 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 06 16:37:08.581178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.581170 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 06 16:37:08.588467 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.588442 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-wprqg"] Jul 06 16:37:08.591227 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.591208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79dzj\" (UniqueName: \"kubernetes.io/projected/320b113e-8b5d-4622-8c88-6f4c3fd28905-kube-api-access-79dzj\") pod \"maas-controller-66fcc46d69-wprqg\" (UID: \"320b113e-8b5d-4622-8c88-6f4c3fd28905\") " pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.591356 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.591276 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/320b113e-8b5d-4622-8c88-6f4c3fd28905-webhook-cert\") pod \"maas-controller-66fcc46d69-wprqg\" (UID: \"320b113e-8b5d-4622-8c88-6f4c3fd28905\") " pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.692781 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.692743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/320b113e-8b5d-4622-8c88-6f4c3fd28905-webhook-cert\") pod \"maas-controller-66fcc46d69-wprqg\" (UID: \"320b113e-8b5d-4622-8c88-6f4c3fd28905\") " pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.692982 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.692862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79dzj\" (UniqueName: \"kubernetes.io/projected/320b113e-8b5d-4622-8c88-6f4c3fd28905-kube-api-access-79dzj\") pod \"maas-controller-66fcc46d69-wprqg\" (UID: \"320b113e-8b5d-4622-8c88-6f4c3fd28905\") " pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.695320 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.695298 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/320b113e-8b5d-4622-8c88-6f4c3fd28905-webhook-cert\") pod \"maas-controller-66fcc46d69-wprqg\" (UID: \"320b113e-8b5d-4622-8c88-6f4c3fd28905\") " pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.703544 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.703521 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79dzj\" (UniqueName: \"kubernetes.io/projected/320b113e-8b5d-4622-8c88-6f4c3fd28905-kube-api-access-79dzj\") pod \"maas-controller-66fcc46d69-wprqg\" (UID: \"320b113e-8b5d-4622-8c88-6f4c3fd28905\") " pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:08.888626 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:08.888584 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:09.027989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:09.027962 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-66fcc46d69-wprqg"] Jul 06 16:37:09.029265 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:37:09.029242 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod320b113e_8b5d_4622_8c88_6f4c3fd28905.slice/crio-6488acaa481b31a5d9febdedfb2c194007f1df62b18d2b15f6915da3344ecb65 WatchSource:0}: Error finding container 6488acaa481b31a5d9febdedfb2c194007f1df62b18d2b15f6915da3344ecb65: Status 404 returned error can't find the container with id 6488acaa481b31a5d9febdedfb2c194007f1df62b18d2b15f6915da3344ecb65 Jul 06 16:37:09.448016 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:09.447981 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fcc46d69-wprqg" event={"ID":"320b113e-8b5d-4622-8c88-6f4c3fd28905","Type":"ContainerStarted","Data":"6488acaa481b31a5d9febdedfb2c194007f1df62b18d2b15f6915da3344ecb65"} Jul 06 16:37:10.454085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:10.454054 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-66fcc46d69-wprqg" event={"ID":"320b113e-8b5d-4622-8c88-6f4c3fd28905","Type":"ContainerStarted","Data":"f444c980b01d55183bc8367a8fd1c2abf7d98b132b4349fd4049e035f70ad1e7"} Jul 06 16:37:10.454483 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:10.454114 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:37:10.481411 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:10.481351 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-66fcc46d69-wprqg" podStartSLOduration=2.022186366 podStartE2EDuration="2.48133326s" totalImagesPullingTime="459.146894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:37:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:37:10.479034144 +0000 UTC m=+813.194253143" watchObservedRunningTime="2026-07-06 16:37:10.48133326 +0000 UTC m=+813.196552234" Jul 06 16:37:21.466833 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:37:21.466774 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-66fcc46d69-wprqg" Jul 06 16:38:37.760316 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:38:37.760226 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:38:37.761718 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:38:37.761698 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:43:37.787486 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:43:37.787455 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:43:37.790150 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:43:37.790124 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:45:00.158177 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.158146 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29722605-mxqnr"] Jul 06 16:45:00.161282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.161265 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:00.164236 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.164216 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-gpfr9\"" Jul 06 16:45:00.171103 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.171077 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722605-mxqnr"] Jul 06 16:45:00.300530 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.300497 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41-kube-api-access-g96jv\") pod \"maas-api-key-cleanup-29722605-mxqnr\" (UID: \"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41\") " pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:00.401250 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.401212 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41-kube-api-access-g96jv\") pod \"maas-api-key-cleanup-29722605-mxqnr\" (UID: \"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41\") " pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:00.411169 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.411116 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41-kube-api-access-g96jv\") pod \"maas-api-key-cleanup-29722605-mxqnr\" (UID: \"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41\") " pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:00.472053 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.472011 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:00.608387 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.608359 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29722605-mxqnr"] Jul 06 16:45:00.610808 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:45:00.610760 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f2d0c15_3f29_4ed1_a31b_6c72c65f9e41.slice/crio-45d96e132474b52b0977b10dd15ad06be04e115f861b003bf823aac05a91fef2 WatchSource:0}: Error finding container 45d96e132474b52b0977b10dd15ad06be04e115f861b003bf823aac05a91fef2: Status 404 returned error can't find the container with id 45d96e132474b52b0977b10dd15ad06be04e115f861b003bf823aac05a91fef2 Jul 06 16:45:00.612723 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.612709 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 16:45:00.720976 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:00.720877 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" event={"ID":"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41","Type":"ContainerStarted","Data":"45d96e132474b52b0977b10dd15ad06be04e115f861b003bf823aac05a91fef2"} Jul 06 16:45:03.737239 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:03.737202 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" event={"ID":"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41","Type":"ContainerStarted","Data":"036f6bc10544245c4e2a77f1582945e5c416c55bd83ac6e87a0e119f5c98631d"} Jul 06 16:45:03.758049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:03.757996 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" podStartSLOduration=0.924829303 podStartE2EDuration="3.757980194s" totalImagesPullingTime="2.833150891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:45:03.755632436 +0000 UTC m=+1286.470851410" watchObservedRunningTime="2026-07-06 16:45:03.757980194 +0000 UTC m=+1286.473199168" Jul 06 16:45:04.743913 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:04.743882 2563 generic.go:356] "Generic (PLEG): container finished" podID="1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41" containerID="036f6bc10544245c4e2a77f1582945e5c416c55bd83ac6e87a0e119f5c98631d" exitCode=0 Jul 06 16:45:04.744313 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:04.743969 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" event={"ID":"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41","Type":"ContainerDied","Data":"036f6bc10544245c4e2a77f1582945e5c416c55bd83ac6e87a0e119f5c98631d"} Jul 06 16:45:05.885027 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:05.885001 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:06.056113 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:06.056030 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41-kube-api-access-g96jv\") pod \"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41\" (UID: \"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41\") " Jul 06 16:45:06.058313 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:06.058274 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41-kube-api-access-g96jv" (OuterVolumeSpecName: "kube-api-access-g96jv") pod "1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41" (UID: "1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41"). InnerVolumeSpecName "kube-api-access-g96jv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:45:06.157459 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:06.157420 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g96jv\" (UniqueName: \"kubernetes.io/projected/1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41-kube-api-access-g96jv\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:45:06.754953 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:06.754918 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" Jul 06 16:45:06.754953 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:06.754941 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29722605-mxqnr" event={"ID":"1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41","Type":"ContainerDied","Data":"45d96e132474b52b0977b10dd15ad06be04e115f861b003bf823aac05a91fef2"} Jul 06 16:45:06.755222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:45:06.754989 2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45d96e132474b52b0977b10dd15ad06be04e115f861b003bf823aac05a91fef2" Jul 06 16:48:37.815386 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:48:37.815357 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:48:37.819190 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:48:37.819172 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:49:40.635587 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.635538 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc"] Jul 06 16:49:40.636176 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.636097 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41" containerName="cleanup" Jul 06 16:49:40.636176 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.636112 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41" containerName="cleanup" Jul 06 16:49:40.636293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.636186 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41" containerName="cleanup" Jul 06 16:49:40.639160 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.639093 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.642258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.642236 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7efb30b2-openshift-default-dockercfg-jkwpl\"" Jul 06 16:49:40.665498 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.665466 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc"] Jul 06 16:49:40.803032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-socket\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803037 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-data\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803283 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803056 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-podinfo\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803283 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803112 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istiod-ca-cert\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803283 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803179 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-token\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803283 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803210 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-envoy\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803283 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgp9k\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-kube-api-access-mgp9k\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803460 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803336 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-credential-socket\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.803460 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.803355 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-certs\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904154 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904071 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-credential-socket\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904154 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904104 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-certs\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904154 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904137 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-socket\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904154 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904157 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-data\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904463 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904175 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-podinfo\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904463 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istiod-ca-cert\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904463 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904242 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-token\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904463 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904272 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-envoy\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904463 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgp9k\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-kube-api-access-mgp9k\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904686 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904516 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-credential-socket\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904686 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904571 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-certs\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904686 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904596 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-socket\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.904686 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.904654 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-data\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.905044 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.905020 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istiod-ca-cert\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.906533 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.906517 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-envoy\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.906837 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.906816 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-podinfo\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.914208 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.914179 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgp9k\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-kube-api-access-mgp9k\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.914626 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.914604 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-token\") pod \"e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:40.949594 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:40.949566 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:41.091209 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:41.091187 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc"] Jul 06 16:49:41.092875 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:49:41.092848 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4332c07c_85e7_4bff_801d_0bff8f84ba5e.slice/crio-8a4f9dc0fa6532eb8477eed23dc15693a62e74d36146903b8d26362309ce91d0 WatchSource:0}: Error finding container 8a4f9dc0fa6532eb8477eed23dc15693a62e74d36146903b8d26362309ce91d0: Status 404 returned error can't find the container with id 8a4f9dc0fa6532eb8477eed23dc15693a62e74d36146903b8d26362309ce91d0 Jul 06 16:49:41.095052 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:41.095008 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:49:41.095159 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:41.095099 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:49:41.095159 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:41.095140 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:49:42.083472 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:42.083437 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" event={"ID":"4332c07c-85e7-4bff-801d-0bff8f84ba5e","Type":"ContainerStarted","Data":"8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722"} Jul 06 16:49:42.083915 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:42.083491 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" event={"ID":"4332c07c-85e7-4bff-801d-0bff8f84ba5e","Type":"ContainerStarted","Data":"8a4f9dc0fa6532eb8477eed23dc15693a62e74d36146903b8d26362309ce91d0"} Jul 06 16:49:42.124825 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:42.124750 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" podStartSLOduration=2.124733118 podStartE2EDuration="2.124733118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:42.123418325 +0000 UTC m=+1564.838637301" watchObservedRunningTime="2026-07-06 16:49:42.124733118 +0000 UTC m=+1564.839952092" Jul 06 16:49:42.950275 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:42.950244 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:42.955319 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:42.955294 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:43.088240 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:43.088211 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:43.089178 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:43.089164 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:47.142877 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:47.142845 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc"] Jul 06 16:49:47.143384 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:47.143033 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" podUID="4332c07c-85e7-4bff-801d-0bff8f84ba5e" containerName="istio-proxy" containerID="cri-o://8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722" gracePeriod=30 Jul 06 16:49:53.678469 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.678419 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh"] Jul 06 16:49:53.684360 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.684334 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.687629 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.687595 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b496212d-openshift-default-dockercfg-vpwm9\"" Jul 06 16:49:53.697312 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.697284 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh"] Jul 06 16:49:53.727291 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727244 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-token\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727302 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-socket\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-certs\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-data\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-podinfo\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727478 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-envoy\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19695a15-34b5-4bcf-aef8-6df2793fef89-istiod-ca-cert\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727580 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-credential-socket\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.727722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.727630 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7pv\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-kube-api-access-st7pv\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828055 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-token\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-socket\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-certs\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828161 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-data\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828214 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-podinfo\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-envoy\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828284 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19695a15-34b5-4bcf-aef8-6df2793fef89-istiod-ca-cert\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828655 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828328 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-credential-socket\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828655 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828368 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7pv\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-kube-api-access-st7pv\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828655 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828585 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-socket\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828849 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-certs\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.828849 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828768 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-data\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.829024 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.828998 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-credential-socket\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.829211 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.829192 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19695a15-34b5-4bcf-aef8-6df2793fef89-istiod-ca-cert\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.830834 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.830779 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-envoy\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.830948 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.830871 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-podinfo\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.838345 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.838319 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-token\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.838821 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.838779 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7pv\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-kube-api-access-st7pv\") pod \"e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:53.999722 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:53.999628 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:54.159637 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.159613 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh"] Jul 06 16:49:54.160211 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:49:54.160181 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19695a15_34b5_4bcf_aef8_6df2793fef89.slice/crio-6253a0aa73d8c7db87b4d33fcbf0e0ef42f24027f6425332e6034887ddec7dbc WatchSource:0}: Error finding container 6253a0aa73d8c7db87b4d33fcbf0e0ef42f24027f6425332e6034887ddec7dbc: Status 404 returned error can't find the container with id 6253a0aa73d8c7db87b4d33fcbf0e0ef42f24027f6425332e6034887ddec7dbc Jul 06 16:49:54.162570 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.162538 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:49:54.162629 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.162606 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:49:54.162664 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.162634 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:49:54.384194 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.384169 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:54.433870 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.433837 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-certs\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.433909 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-token\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.433965 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-podinfo\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.433991 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istiod-ca-cert\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434023 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-envoy\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434060 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434049 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgp9k\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-kube-api-access-mgp9k\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434320 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434111 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-socket\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434320 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434136 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:49:54.434320 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434146 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-data\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434320 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434210 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-credential-socket\") pod \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\" (UID: \"4332c07c-85e7-4bff-801d-0bff8f84ba5e\") " Jul 06 16:49:54.434524 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434343 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:49:54.434578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434518 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:49:54.434578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434570 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.434682 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434591 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.435207 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434746 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:49:54.435207 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.434932 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-data" (OuterVolumeSpecName: "istio-data") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:49:54.436674 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.436654 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-token" (OuterVolumeSpecName: "istio-token") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:49:54.436870 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.436853 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-kube-api-access-mgp9k" (OuterVolumeSpecName: "kube-api-access-mgp9k") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "kube-api-access-mgp9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:49:54.436933 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.436892 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:49:54.437132 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.437116 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4332c07c-85e7-4bff-801d-0bff8f84ba5e" (UID: "4332c07c-85e7-4bff-801d-0bff8f84ba5e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535000 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535033 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535048 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535059 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535070 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535081 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4332c07c-85e7-4bff-801d-0bff8f84ba5e-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:54.535098 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:54.535093 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgp9k\" (UniqueName: \"kubernetes.io/projected/4332c07c-85e7-4bff-801d-0bff8f84ba5e-kube-api-access-mgp9k\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:49:55.152848 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.152744 2563 generic.go:356] "Generic (PLEG): container finished" podID="4332c07c-85e7-4bff-801d-0bff8f84ba5e" containerID="8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722" exitCode=0 Jul 06 16:49:55.152848 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.152802 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" Jul 06 16:49:55.152848 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.152820 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" event={"ID":"4332c07c-85e7-4bff-801d-0bff8f84ba5e","Type":"ContainerDied","Data":"8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722"} Jul 06 16:49:55.153367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.152869 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc" event={"ID":"4332c07c-85e7-4bff-801d-0bff8f84ba5e","Type":"ContainerDied","Data":"8a4f9dc0fa6532eb8477eed23dc15693a62e74d36146903b8d26362309ce91d0"} Jul 06 16:49:55.153367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.152881 2563 scope.go:117] "RemoveContainer" containerID="8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722" Jul 06 16:49:55.155163 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.154818 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" event={"ID":"19695a15-34b5-4bcf-aef8-6df2793fef89","Type":"ContainerStarted","Data":"5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9"} Jul 06 16:49:55.155163 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.154873 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" event={"ID":"19695a15-34b5-4bcf-aef8-6df2793fef89","Type":"ContainerStarted","Data":"6253a0aa73d8c7db87b4d33fcbf0e0ef42f24027f6425332e6034887ddec7dbc"} Jul 06 16:49:55.166403 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.166323 2563 scope.go:117] "RemoveContainer" containerID="8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722" Jul 06 16:49:55.166749 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:49:55.166721 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722\": container with ID starting with 8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722 not found: ID does not exist" containerID="8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722" Jul 06 16:49:55.167016 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.166760 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722"} err="failed to get container status \"8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722\": rpc error: code = NotFound desc = could not find container \"8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722\": container with ID starting with 8e7de9dd23da0b74e9a6370477af2c0ed204d6b34bedef817041fbabb5f8b722 not found: ID does not exist" Jul 06 16:49:55.187136 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.187059 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" podStartSLOduration=2.187032077 podStartE2EDuration="2.187032077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:49:55.185367521 +0000 UTC m=+1577.900586496" watchObservedRunningTime="2026-07-06 16:49:55.187032077 +0000 UTC m=+1577.902251051" Jul 06 16:49:55.204220 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.204183 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc"] Jul 06 16:49:55.211462 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.211433 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7efb30b2-openshift-default-6987dfdb89-8lkzc"] Jul 06 16:49:55.793390 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:55.793356 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4332c07c-85e7-4bff-801d-0bff8f84ba5e" path="/var/lib/kubelet/pods/4332c07c-85e7-4bff-801d-0bff8f84ba5e/volumes" Jul 06 16:49:56.000863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:56.000829 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:56.005758 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:56.005733 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:56.160500 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:56.160461 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:49:56.161336 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:49:56.161316 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:50:06.209425 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.209396 2563 generic.go:356] "Generic (PLEG): container finished" podID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerID="5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9" exitCode=0 Jul 06 16:50:06.209872 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.209475 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" event={"ID":"19695a15-34b5-4bcf-aef8-6df2793fef89","Type":"ContainerDied","Data":"5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9"} Jul 06 16:50:06.209998 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.209981 2563 scope.go:117] "RemoveContainer" containerID="5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9" Jul 06 16:50:06.211084 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.211055 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:50:06.211192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.211143 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:50:06.211253 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.211190 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:50:06.313408 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:06.313377 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh"] Jul 06 16:50:07.216510 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:07.216473 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" event={"ID":"19695a15-34b5-4bcf-aef8-6df2793fef89","Type":"ContainerStarted","Data":"9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361"} Jul 06 16:50:07.216955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:07.216534 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" containerID="cri-o://9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361" gracePeriod=30 Jul 06 16:50:11.161367 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:11.161329 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:50:12.473135 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.473072 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:50:12.616321 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616289 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-token\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616332 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-socket\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616356 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-podinfo\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616371 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-credential-socket\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616394 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-data\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616462 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-certs\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616477 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19695a15-34b5-4bcf-aef8-6df2793fef89-istiod-ca-cert\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616532 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7pv\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-kube-api-access-st7pv\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616583 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-envoy\") pod \"19695a15-34b5-4bcf-aef8-6df2793fef89\" (UID: \"19695a15-34b5-4bcf-aef8-6df2793fef89\") " Jul 06 16:50:12.616955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616713 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:12.616955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616771 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:12.616955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.616836 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:12.617159 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.617016 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.617159 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.617035 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.617159 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.617050 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.617159 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.617089 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19695a15-34b5-4bcf-aef8-6df2793fef89-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:50:12.617415 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.617388 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-data" (OuterVolumeSpecName: "istio-data") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:12.618602 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.618570 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-token" (OuterVolumeSpecName: "istio-token") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:50:12.618763 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.618748 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:12.619013 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.618756 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:50:12.619013 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.618838 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-kube-api-access-st7pv" (OuterVolumeSpecName: "kube-api-access-st7pv") pod "19695a15-34b5-4bcf-aef8-6df2793fef89" (UID: "19695a15-34b5-4bcf-aef8-6df2793fef89"). InnerVolumeSpecName "kube-api-access-st7pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:50:12.717917 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.717883 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.717917 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.717914 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.717917 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.717927 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.718155 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.717936 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19695a15-34b5-4bcf-aef8-6df2793fef89-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.718155 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.717945 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19695a15-34b5-4bcf-aef8-6df2793fef89-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.718155 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.717953 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7pv\" (UniqueName: \"kubernetes.io/projected/19695a15-34b5-4bcf-aef8-6df2793fef89-kube-api-access-st7pv\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:12.834456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834364 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq"] Jul 06 16:50:12.834772 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834755 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" Jul 06 16:50:12.834772 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834771 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" Jul 06 16:50:12.834962 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834805 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4332c07c-85e7-4bff-801d-0bff8f84ba5e" containerName="istio-proxy" Jul 06 16:50:12.834962 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834812 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4332c07c-85e7-4bff-801d-0bff8f84ba5e" containerName="istio-proxy" Jul 06 16:50:12.834962 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834921 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" Jul 06 16:50:12.834962 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834931 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" Jul 06 16:50:12.834962 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.834941 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4332c07c-85e7-4bff-801d-0bff8f84ba5e" containerName="istio-proxy" Jul 06 16:50:12.838443 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.838422 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.841632 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.841611 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-39de03b3-openshift-default-dockercfg-nkq4r\"" Jul 06 16:50:12.851819 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.851760 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq"] Jul 06 16:50:12.919770 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.919734 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-certs\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.919989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.919842 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-socket\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.919989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.919860 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-data\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.919989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.919907 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-envoy\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.919989 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.919950 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istiod-ca-cert\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.920212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.920051 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st4gz\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-kube-api-access-st4gz\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.920212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.920085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-token\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.920212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.920123 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-podinfo\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:12.920212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:12.920188 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-credential-socket\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021072 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021030 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-socket\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021072 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021070 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-data\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021314 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021114 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-envoy\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021314 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021138 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istiod-ca-cert\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021314 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st4gz\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-kube-api-access-st4gz\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021314 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021223 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-token\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021314 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021282 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-podinfo\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021624 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021325 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-credential-socket\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021624 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021361 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-certs\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021624 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021478 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-socket\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021777 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021728 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-certs\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021858 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021810 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-credential-socket\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.021988 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.021971 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-data\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.022239 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.022211 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istiod-ca-cert\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.023351 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.023329 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-envoy\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.023859 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.023841 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-podinfo\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.038147 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.038114 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-token\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.038993 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.038957 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st4gz\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-kube-api-access-st4gz\") pod \"e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.151617 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.151585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:13.251035 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.250994 2563 generic.go:356] "Generic (PLEG): container finished" podID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerID="9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361" exitCode=0 Jul 06 16:50:13.251235 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.251050 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" Jul 06 16:50:13.251235 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.251081 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" event={"ID":"19695a15-34b5-4bcf-aef8-6df2793fef89","Type":"ContainerDied","Data":"9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361"} Jul 06 16:50:13.251235 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.251145 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh" event={"ID":"19695a15-34b5-4bcf-aef8-6df2793fef89","Type":"ContainerDied","Data":"6253a0aa73d8c7db87b4d33fcbf0e0ef42f24027f6425332e6034887ddec7dbc"} Jul 06 16:50:13.251235 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.251180 2563 scope.go:117] "RemoveContainer" containerID="9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361" Jul 06 16:50:13.270542 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.270520 2563 scope.go:117] "RemoveContainer" containerID="5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9" Jul 06 16:50:13.281737 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.281711 2563 scope.go:117] "RemoveContainer" containerID="9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361" Jul 06 16:50:13.282111 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:50:13.282084 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361\": container with ID starting with 9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361 not found: ID does not exist" containerID="9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361" Jul 06 16:50:13.282162 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.282123 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361"} err="failed to get container status \"9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361\": rpc error: code = NotFound desc = could not find container \"9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361\": container with ID starting with 9ea433f8bfc8f81cf7d9bc68de1ae4bdbc7c5988e5e11334e6a91279dfb77361 not found: ID does not exist" Jul 06 16:50:13.282162 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.282143 2563 scope.go:117] "RemoveContainer" containerID="5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9" Jul 06 16:50:13.282463 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:50:13.282444 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9\": container with ID starting with 5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9 not found: ID does not exist" containerID="5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9" Jul 06 16:50:13.282509 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.282472 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9"} err="failed to get container status \"5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9\": rpc error: code = NotFound desc = could not find container \"5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9\": container with ID starting with 5702dd3de76acf51659459c8e6665b645b5a6f08eb4dc2568d3eedecadf7a3d9 not found: ID does not exist" Jul 06 16:50:13.288831 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.288804 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh"] Jul 06 16:50:13.296807 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.296762 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b496212d-openshift-default-6879c7bc59-vh7lh"] Jul 06 16:50:13.310709 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.310673 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq"] Jul 06 16:50:13.321509 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.321465 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:50:13.321738 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.321719 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:50:13.321900 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.321883 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:50:13.794146 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:13.794112 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" path="/var/lib/kubelet/pods/19695a15-34b5-4bcf-aef8-6df2793fef89/volumes" Jul 06 16:50:14.260101 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:14.260064 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" event={"ID":"f7f6a1c5-3be7-4b91-97a0-e79df3d13631","Type":"ContainerStarted","Data":"79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087"} Jul 06 16:50:14.260307 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:14.260130 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" event={"ID":"f7f6a1c5-3be7-4b91-97a0-e79df3d13631","Type":"ContainerStarted","Data":"26d64b2fe14e953c956c8ecc24551f66d1d68b6d5669da8987cb588aa2009809"} Jul 06 16:50:14.300029 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:14.299978 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" podStartSLOduration=2.299959681 podStartE2EDuration="2.299959681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:50:14.297940298 +0000 UTC m=+1597.013159291" watchObservedRunningTime="2026-07-06 16:50:14.299959681 +0000 UTC m=+1597.015178656" Jul 06 16:50:15.152153 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:15.152124 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:15.156886 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:15.156860 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:15.266180 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:15.266139 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:15.266997 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:15.266976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:18.749609 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:18.749573 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq"] Jul 06 16:50:18.750165 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:18.749786 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" podUID="f7f6a1c5-3be7-4b91-97a0-e79df3d13631" containerName="istio-proxy" containerID="cri-o://79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087" gracePeriod=30 Jul 06 16:50:24.005411 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.005384 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:24.129939 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.129901 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-certs\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.129939 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.129946 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-envoy\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130189 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.129974 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-credential-socket\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130189 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130012 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-data\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130189 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130071 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-token\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130189 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130121 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-podinfo\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130189 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130152 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-socket\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130221 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istiod-ca-cert\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130248 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:24.130456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130285 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st4gz\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-kube-api-access-st4gz\") pod \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\" (UID: \"f7f6a1c5-3be7-4b91-97a0-e79df3d13631\") " Jul 06 16:50:24.130456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130342 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:24.130456 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130380 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-data" (OuterVolumeSpecName: "istio-data") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:24.130708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130457 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:24.130708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130620 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:50:24.130708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130642 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.130708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130664 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.130708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130679 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.130708 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.130693 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.132284 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.132252 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:50:24.132408 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.132324 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-kube-api-access-st4gz" (OuterVolumeSpecName: "kube-api-access-st4gz") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "kube-api-access-st4gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:50:24.132478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.132405 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:50:24.132535 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.132508 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-token" (OuterVolumeSpecName: "istio-token") pod "f7f6a1c5-3be7-4b91-97a0-e79df3d13631" (UID: "f7f6a1c5-3be7-4b91-97a0-e79df3d13631"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:50:24.231674 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.231643 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st4gz\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-kube-api-access-st4gz\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.231674 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.231672 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.231674 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.231684 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.231943 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.231692 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.231943 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.231700 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f7f6a1c5-3be7-4b91-97a0-e79df3d13631-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:50:24.312853 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.312819 2563 generic.go:356] "Generic (PLEG): container finished" podID="f7f6a1c5-3be7-4b91-97a0-e79df3d13631" containerID="79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087" exitCode=0 Jul 06 16:50:24.312853 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.312845 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" Jul 06 16:50:24.313072 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.312884 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" event={"ID":"f7f6a1c5-3be7-4b91-97a0-e79df3d13631","Type":"ContainerDied","Data":"79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087"} Jul 06 16:50:24.313072 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.312940 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq" event={"ID":"f7f6a1c5-3be7-4b91-97a0-e79df3d13631","Type":"ContainerDied","Data":"26d64b2fe14e953c956c8ecc24551f66d1d68b6d5669da8987cb588aa2009809"} Jul 06 16:50:24.313072 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.312964 2563 scope.go:117] "RemoveContainer" containerID="79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087" Jul 06 16:50:24.322320 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.322305 2563 scope.go:117] "RemoveContainer" containerID="79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087" Jul 06 16:50:24.322572 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:50:24.322555 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087\": container with ID starting with 79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087 not found: ID does not exist" containerID="79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087" Jul 06 16:50:24.322614 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.322579 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087"} err="failed to get container status \"79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087\": rpc error: code = NotFound desc = could not find container \"79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087\": container with ID starting with 79e2f26638482f0e6ca302b305dd9495d254e7b4aa9230fcb5d77656db99a087 not found: ID does not exist" Jul 06 16:50:24.336786 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.336763 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq"] Jul 06 16:50:24.341256 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:24.341236 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-39de03b3-openshift-default-6dd667d95f-gz5pq"] Jul 06 16:50:25.793321 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:50:25.793289 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7f6a1c5-3be7-4b91-97a0-e79df3d13631" path="/var/lib/kubelet/pods/f7f6a1c5-3be7-4b91-97a0-e79df3d13631/volumes" Jul 06 16:52:03.983221 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.983137 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9"] Jul 06 16:52:03.983701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.983506 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7f6a1c5-3be7-4b91-97a0-e79df3d13631" containerName="istio-proxy" Jul 06 16:52:03.983701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.983517 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7f6a1c5-3be7-4b91-97a0-e79df3d13631" containerName="istio-proxy" Jul 06 16:52:03.983701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.983532 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" Jul 06 16:52:03.983701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.983537 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="19695a15-34b5-4bcf-aef8-6df2793fef89" containerName="istio-proxy" Jul 06 16:52:03.983701 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.983602 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7f6a1c5-3be7-4b91-97a0-e79df3d13631" containerName="istio-proxy" Jul 06 16:52:03.986621 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.986606 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:03.989610 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:03.989587 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1e5ead78-openshift-default-dockercfg-4crml\"" Jul 06 16:52:04.017342 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.017305 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9"] Jul 06 16:52:04.034829 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.034781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-certs\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.034974 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.034845 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-socket\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.034974 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.034910 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840686a1-bd01-4275-a588-d6b3274a0292-istio-podinfo\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.034974 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.034937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840686a1-bd01-4275-a588-d6b3274a0292-istiod-ca-cert\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.034974 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.034958 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-istio-token\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.035110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.034980 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2lv4\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-kube-api-access-t2lv4\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.035110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.035048 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-credential-socket\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.035110 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.035086 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-data\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.035206 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.035110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-envoy\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136336 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136297 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840686a1-bd01-4275-a588-d6b3274a0292-istio-podinfo\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136523 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136347 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840686a1-bd01-4275-a588-d6b3274a0292-istiod-ca-cert\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136523 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136387 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-istio-token\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136523 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136410 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2lv4\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-kube-api-access-t2lv4\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136523 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136466 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-credential-socket\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136523 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-data\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136830 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-envoy\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136830 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136646 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-certs\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.136830 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136708 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-socket\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.137007 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.136987 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-credential-socket\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.137056 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.137017 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-data\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.137056 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.137039 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-socket\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.137123 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.137077 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-certs\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.137428 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.137405 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840686a1-bd01-4275-a588-d6b3274a0292-istiod-ca-cert\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.138887 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.138864 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-envoy\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.139003 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.138982 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840686a1-bd01-4275-a588-d6b3274a0292-istio-podinfo\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.148317 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.148289 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2lv4\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-kube-api-access-t2lv4\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.150618 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.150591 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-istio-token\") pod \"e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.296995 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.296893 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:04.446719 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.446636 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9"] Jul 06 16:52:04.449511 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:52:04.449486 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod840686a1_bd01_4275_a588_d6b3274a0292.slice/crio-7ef45c807e7b653dc0ff4d648498cf51fc01a7868fc2cec4b0c6bc588ec8df9c WatchSource:0}: Error finding container 7ef45c807e7b653dc0ff4d648498cf51fc01a7868fc2cec4b0c6bc588ec8df9c: Status 404 returned error can't find the container with id 7ef45c807e7b653dc0ff4d648498cf51fc01a7868fc2cec4b0c6bc588ec8df9c Jul 06 16:52:04.451585 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.451550 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:04.451638 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.451626 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:04.451674 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.451656 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:04.817647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.817601 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" event={"ID":"840686a1-bd01-4275-a588-d6b3274a0292","Type":"ContainerStarted","Data":"93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020"} Jul 06 16:52:04.817854 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.817700 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" event={"ID":"840686a1-bd01-4275-a588-d6b3274a0292","Type":"ContainerStarted","Data":"7ef45c807e7b653dc0ff4d648498cf51fc01a7868fc2cec4b0c6bc588ec8df9c"} Jul 06 16:52:04.844525 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:04.844475 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" podStartSLOduration=1.8444550899999999 podStartE2EDuration="1.84445509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:04.843444305 +0000 UTC m=+1707.558663338" watchObservedRunningTime="2026-07-06 16:52:04.84445509 +0000 UTC m=+1707.559674064" Jul 06 16:52:05.297982 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:05.297943 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:05.303252 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:05.303222 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:05.823759 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:05.823726 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:05.824685 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:05.824669 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:19.899562 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:19.899524 2563 generic.go:356] "Generic (PLEG): container finished" podID="840686a1-bd01-4275-a588-d6b3274a0292" containerID="93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020" exitCode=0 Jul 06 16:52:19.900043 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:19.899599 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" event={"ID":"840686a1-bd01-4275-a588-d6b3274a0292","Type":"ContainerDied","Data":"93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020"} Jul 06 16:52:19.900043 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:19.900028 2563 scope.go:117] "RemoveContainer" containerID="93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020" Jul 06 16:52:19.900988 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:19.900960 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:19.901049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:19.901027 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:19.901091 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:19.901065 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:20.824374 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:20.824336 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:20.910427 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:20.909825 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" event={"ID":"840686a1-bd01-4275-a588-d6b3274a0292","Type":"ContainerStarted","Data":"6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81"} Jul 06 16:52:21.297699 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:21.297667 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:21.298905 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:21.298879 2563 patch_prober.go:28] interesting pod/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 16:52:21.299112 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:21.299084 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 16:52:22.298319 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:22.298281 2563 patch_prober.go:28] interesting pod/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 16:52:22.298955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:22.298921 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 16:52:23.298305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:23.298261 2563 patch_prober.go:28] interesting pod/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 16:52:23.298521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:23.298327 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 16:52:24.297710 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:24.297662 2563 patch_prober.go:28] interesting pod/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 06 16:52:24.297957 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:24.297734 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 06 16:52:25.301029 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:25.301003 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:25.301450 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:25.301398 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:25.301993 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:25.301979 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:26.432827 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:26.432778 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9"] Jul 06 16:52:26.941619 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:26.941586 2563 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" secret="" err="secret \"e2e-mt-1e5ead78-openshift-default-dockercfg-4crml\" not found" Jul 06 16:52:27.945656 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:27.945617 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" containerID="cri-o://6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81" gracePeriod=30 Jul 06 16:52:33.207118 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.207053 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:33.322023 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.321986 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-istio-token\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322023 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322027 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-credential-socket\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322264 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322073 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840686a1-bd01-4275-a588-d6b3274a0292-istio-podinfo\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322264 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322102 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-certs\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322264 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322123 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-data\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322264 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322164 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-envoy\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322264 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322198 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-socket\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322264 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322255 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2lv4\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-kube-api-access-t2lv4\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322509 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840686a1-bd01-4275-a588-d6b3274a0292-istiod-ca-cert\") pod \"840686a1-bd01-4275-a588-d6b3274a0292\" (UID: \"840686a1-bd01-4275-a588-d6b3274a0292\") " Jul 06 16:52:33.322509 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322494 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:52:33.322621 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322580 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.322810 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322757 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:52:33.322891 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322844 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/840686a1-bd01-4275-a588-d6b3274a0292-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:52:33.322976 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.322948 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:52:33.323372 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.323350 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-data" (OuterVolumeSpecName: "istio-data") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:52:33.324417 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.324378 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/840686a1-bd01-4275-a588-d6b3274a0292-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:52:33.324696 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.324663 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-kube-api-access-t2lv4" (OuterVolumeSpecName: "kube-api-access-t2lv4") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "kube-api-access-t2lv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:52:33.324925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.324736 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:52:33.324925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.324746 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-istio-token" (OuterVolumeSpecName: "istio-token") pod "840686a1-bd01-4275-a588-d6b3274a0292" (UID: "840686a1-bd01-4275-a588-d6b3274a0292"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:52:33.423662 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423628 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423662 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423656 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/840686a1-bd01-4275-a588-d6b3274a0292-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423662 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423671 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423683 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423690 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/840686a1-bd01-4275-a588-d6b3274a0292-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423700 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2lv4\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-kube-api-access-t2lv4\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423709 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/840686a1-bd01-4275-a588-d6b3274a0292-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.423927 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.423717 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/840686a1-bd01-4275-a588-d6b3274a0292-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:52:33.973882 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.973847 2563 generic.go:356] "Generic (PLEG): container finished" podID="840686a1-bd01-4275-a588-d6b3274a0292" containerID="6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81" exitCode=0 Jul 06 16:52:33.974071 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.973894 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" Jul 06 16:52:33.974071 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.973928 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" event={"ID":"840686a1-bd01-4275-a588-d6b3274a0292","Type":"ContainerDied","Data":"6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81"} Jul 06 16:52:33.974071 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.973984 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9" event={"ID":"840686a1-bd01-4275-a588-d6b3274a0292","Type":"ContainerDied","Data":"7ef45c807e7b653dc0ff4d648498cf51fc01a7868fc2cec4b0c6bc588ec8df9c"} Jul 06 16:52:33.974071 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.974012 2563 scope.go:117] "RemoveContainer" containerID="6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81" Jul 06 16:52:33.982850 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.982832 2563 scope.go:117] "RemoveContainer" containerID="93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020" Jul 06 16:52:33.990925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.990903 2563 scope.go:117] "RemoveContainer" containerID="6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81" Jul 06 16:52:33.991206 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:52:33.991187 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81\": container with ID starting with 6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81 not found: ID does not exist" containerID="6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81" Jul 06 16:52:33.991252 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.991216 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81"} err="failed to get container status \"6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81\": rpc error: code = NotFound desc = could not find container \"6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81\": container with ID starting with 6d45fa471460a45ad74b8d2576350bc7ad23f755d01ec4363c01fab1539dfc81 not found: ID does not exist" Jul 06 16:52:33.991252 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.991235 2563 scope.go:117] "RemoveContainer" containerID="93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020" Jul 06 16:52:33.991455 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:52:33.991441 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020\": container with ID starting with 93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020 not found: ID does not exist" containerID="93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020" Jul 06 16:52:33.991496 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.991458 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020"} err="failed to get container status \"93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020\": rpc error: code = NotFound desc = could not find container \"93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020\": container with ID starting with 93c2180bd59e79238b72cd5498f16b274d46fa8a5d579d4e27b4586c3d57c020 not found: ID does not exist" Jul 06 16:52:33.999175 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:33.999148 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9"] Jul 06 16:52:34.005056 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:34.005031 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1e5ead78-openshift-default-7f47468699-knnh9"] Jul 06 16:52:35.793855 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:35.793825 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="840686a1-bd01-4275-a588-d6b3274a0292" path="/var/lib/kubelet/pods/840686a1-bd01-4275-a588-d6b3274a0292/volumes" Jul 06 16:52:47.434748 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.434704 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r"] Jul 06 16:52:47.435258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.435128 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" Jul 06 16:52:47.435258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.435141 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" Jul 06 16:52:47.435258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.435151 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" Jul 06 16:52:47.435258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.435157 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" Jul 06 16:52:47.435258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.435219 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" Jul 06 16:52:47.435258 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.435229 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="840686a1-bd01-4275-a588-d6b3274a0292" containerName="istio-proxy" Jul 06 16:52:47.438315 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.438299 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.441206 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.441184 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8f668f5c-openshift-default-dockercfg-wz5b6\"" Jul 06 16:52:47.449639 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.449613 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r"] Jul 06 16:52:47.455667 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455642 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-credential-socket\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.455840 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455682 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-certs\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.455840 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455700 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-data\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.455840 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455777 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2937772-32a1-41a7-82be-2aa60e232732-istio-podinfo\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.455958 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455857 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lgw\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-kube-api-access-86lgw\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.455958 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455891 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-socket\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.455958 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2937772-32a1-41a7-82be-2aa60e232732-istiod-ca-cert\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.456101 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.455956 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-envoy\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.456101 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.456001 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-istio-token\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557302 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-envoy\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557339 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-istio-token\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557590 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557384 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-credential-socket\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557590 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557412 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-certs\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557590 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557427 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-data\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557586 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2937772-32a1-41a7-82be-2aa60e232732-istio-podinfo\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557639 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lgw\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-kube-api-access-86lgw\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557684 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-socket\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557712 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2937772-32a1-41a7-82be-2aa60e232732-istiod-ca-cert\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557993 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557845 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-data\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.557993 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.557869 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-credential-socket\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.558222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.558201 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-certs\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.558300 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.558277 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-socket\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.558345 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.558308 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2937772-32a1-41a7-82be-2aa60e232732-istiod-ca-cert\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.559713 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.559691 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-envoy\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.560117 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.560098 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2937772-32a1-41a7-82be-2aa60e232732-istio-podinfo\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.566563 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.566533 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-istio-token\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.566738 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.566718 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lgw\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-kube-api-access-86lgw\") pod \"e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.749361 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.749271 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:47.896089 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.896063 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r"] Jul 06 16:52:47.898279 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:52:47.898241 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2937772_32a1_41a7_82be_2aa60e232732.slice/crio-b1b837e74421494fdb6bb84d09881e35735036521b00669b32248094ef7cf0f0 WatchSource:0}: Error finding container b1b837e74421494fdb6bb84d09881e35735036521b00669b32248094ef7cf0f0: Status 404 returned error can't find the container with id b1b837e74421494fdb6bb84d09881e35735036521b00669b32248094ef7cf0f0 Jul 06 16:52:47.900511 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.900472 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:47.900612 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.900543 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:47.900612 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:47.900586 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:52:48.049932 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:48.049831 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" event={"ID":"f2937772-32a1-41a7-82be-2aa60e232732","Type":"ContainerStarted","Data":"64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9"} Jul 06 16:52:48.049932 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:48.049908 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" event={"ID":"f2937772-32a1-41a7-82be-2aa60e232732","Type":"ContainerStarted","Data":"b1b837e74421494fdb6bb84d09881e35735036521b00669b32248094ef7cf0f0"} Jul 06 16:52:48.075735 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:48.075673 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" podStartSLOduration=1.075656808 podStartE2EDuration="1.075656808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:52:48.074704696 +0000 UTC m=+1750.789923670" watchObservedRunningTime="2026-07-06 16:52:48.075656808 +0000 UTC m=+1750.790875781" Jul 06 16:52:48.749453 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:48.749422 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:48.761565 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:48.761536 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:49.055055 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:49.054958 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:52:49.055910 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:52:49.055891 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:53:03.135283 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:03.135254 2563 generic.go:356] "Generic (PLEG): container finished" podID="f2937772-32a1-41a7-82be-2aa60e232732" containerID="64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9" exitCode=0 Jul 06 16:53:03.135705 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:03.135332 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" event={"ID":"f2937772-32a1-41a7-82be-2aa60e232732","Type":"ContainerDied","Data":"64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9"} Jul 06 16:53:03.135761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:03.135750 2563 scope.go:117] "RemoveContainer" containerID="64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9" Jul 06 16:53:03.136806 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:03.136766 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:53:03.136923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:03.136852 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:53:03.136923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:03.136884 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:53:04.055548 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:04.055497 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:53:04.142835 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:04.142787 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" event={"ID":"f2937772-32a1-41a7-82be-2aa60e232732","Type":"ContainerStarted","Data":"695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc"} Jul 06 16:53:04.749485 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:04.749441 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:53:04.750578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:04.750551 2563 patch_prober.go:28] interesting pod/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 16:53:04.750642 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:04.750617 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 16:53:05.750581 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:05.750543 2563 patch_prober.go:28] interesting pod/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 16:53:05.751094 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:05.750607 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 16:53:06.750246 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:06.750205 2563 patch_prober.go:28] interesting pod/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 16:53:06.750436 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:06.750262 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 16:53:07.750173 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:07.750132 2563 patch_prober.go:28] interesting pod/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Jul 06 16:53:07.750565 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:07.750203 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Jul 06 16:53:08.753731 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:08.753700 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:53:08.754132 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:08.754013 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:53:08.754649 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:08.754626 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:53:37.845019 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:37.844992 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:53:37.849168 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:53:37.849149 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:54:45.315299 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:45.315267 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r"] Jul 06 16:54:45.315853 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:45.315492 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" containerID="cri-o://695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc" gracePeriod=30 Jul 06 16:54:50.581215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.581151 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:54:50.675940 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.675906 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2937772-32a1-41a7-82be-2aa60e232732-istio-podinfo\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.675940 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.675943 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2937772-32a1-41a7-82be-2aa60e232732-istiod-ca-cert\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676193 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.675994 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-socket\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676193 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676087 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-credential-socket\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676193 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676121 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-86lgw\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-kube-api-access-86lgw\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676193 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676149 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-envoy\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676193 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-istio-token\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676474 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676202 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-certs\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676474 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676242 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-data\") pod \"f2937772-32a1-41a7-82be-2aa60e232732\" (UID: \"f2937772-32a1-41a7-82be-2aa60e232732\") " Jul 06 16:54:50.676474 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676315 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:54:50.676474 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676361 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:54:50.676474 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676382 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2937772-32a1-41a7-82be-2aa60e232732-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:54:50.676741 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676600 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:54:50.676741 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676680 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.676741 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676707 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.676741 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676726 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2937772-32a1-41a7-82be-2aa60e232732-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.676968 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.676742 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.677340 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.677314 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-data" (OuterVolumeSpecName: "istio-data") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:54:50.678370 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.678343 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2937772-32a1-41a7-82be-2aa60e232732-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:54:50.678928 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.678903 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-kube-api-access-86lgw" (OuterVolumeSpecName: "kube-api-access-86lgw") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "kube-api-access-86lgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:54:50.679017 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.678942 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:54:50.679017 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.678955 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-istio-token" (OuterVolumeSpecName: "istio-token") pod "f2937772-32a1-41a7-82be-2aa60e232732" (UID: "f2937772-32a1-41a7-82be-2aa60e232732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:54:50.684224 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.684198 2563 generic.go:356] "Generic (PLEG): container finished" podID="f2937772-32a1-41a7-82be-2aa60e232732" containerID="695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc" exitCode=0 Jul 06 16:54:50.684333 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.684264 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" Jul 06 16:54:50.684333 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.684274 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" event={"ID":"f2937772-32a1-41a7-82be-2aa60e232732","Type":"ContainerDied","Data":"695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc"} Jul 06 16:54:50.684333 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.684329 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r" event={"ID":"f2937772-32a1-41a7-82be-2aa60e232732","Type":"ContainerDied","Data":"b1b837e74421494fdb6bb84d09881e35735036521b00669b32248094ef7cf0f0"} Jul 06 16:54:50.684484 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.684352 2563 scope.go:117] "RemoveContainer" containerID="695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc" Jul 06 16:54:50.699692 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.699668 2563 scope.go:117] "RemoveContainer" containerID="64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9" Jul 06 16:54:50.709861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.709838 2563 scope.go:117] "RemoveContainer" containerID="695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc" Jul 06 16:54:50.710197 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:54:50.710174 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc\": container with ID starting with 695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc not found: ID does not exist" containerID="695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc" Jul 06 16:54:50.710263 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.710215 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc"} err="failed to get container status \"695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc\": rpc error: code = NotFound desc = could not find container \"695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc\": container with ID starting with 695adb7b3091ee40e515e7fd0d50714aae144a9759d12ae357973262508b3ecc not found: ID does not exist" Jul 06 16:54:50.710263 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.710241 2563 scope.go:117] "RemoveContainer" containerID="64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9" Jul 06 16:54:50.710544 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:54:50.710527 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9\": container with ID starting with 64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9 not found: ID does not exist" containerID="64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9" Jul 06 16:54:50.710599 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.710555 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9"} err="failed to get container status \"64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9\": rpc error: code = NotFound desc = could not find container \"64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9\": container with ID starting with 64912db1d329413933121ba0ef1c93bed4a05fb15b41793be338099c6e9829e9 not found: ID does not exist" Jul 06 16:54:50.713311 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.713288 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r"] Jul 06 16:54:50.719838 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.719812 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8f668f5c-openshift-default-84db676787-dfq7r"] Jul 06 16:54:50.778039 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.778012 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.778039 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.778036 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2937772-32a1-41a7-82be-2aa60e232732-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.778203 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.778045 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86lgw\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-kube-api-access-86lgw\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.778203 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.778057 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2937772-32a1-41a7-82be-2aa60e232732-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:50.778203 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:50.778066 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2937772-32a1-41a7-82be-2aa60e232732-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:54:51.793224 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:54:51.793190 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2937772-32a1-41a7-82be-2aa60e232732" path="/var/lib/kubelet/pods/f2937772-32a1-41a7-82be-2aa60e232732/volumes" Jul 06 16:55:43.576481 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.576449 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6"] Jul 06 16:55:43.576923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.576889 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" Jul 06 16:55:43.576923 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.576902 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" Jul 06 16:55:43.577014 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.576980 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" Jul 06 16:55:43.577014 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.576987 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" Jul 06 16:55:43.579978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.579961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.583615 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.583595 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-da00b0-openshift-default-dockercfg-t5djs\"" Jul 06 16:55:43.594823 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.594778 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6"] Jul 06 16:55:43.749222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-socket\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749223 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-envoy\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749246 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-credential-socket\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-data\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749317 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-certs\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749381 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-podinfo\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749440 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749421 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-token\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749602 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749442 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vg7qt\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-kube-api-access-vg7qt\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.749602 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.749468 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda41d0b-8915-4449-bcb4-549a191f3b20-istiod-ca-cert\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.850850 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.850786 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-socket\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851028 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.850944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-envoy\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851028 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.850984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-credential-socket\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851028 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-data\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851056 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-certs\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851099 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-podinfo\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-token\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851360 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851178 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vg7qt\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-kube-api-access-vg7qt\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851360 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851187 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-socket\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851360 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851211 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda41d0b-8915-4449-bcb4-549a191f3b20-istiod-ca-cert\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851400 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-data\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851576 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851546 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-certs\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851647 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851624 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-credential-socket\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.851878 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.851858 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda41d0b-8915-4449-bcb4-549a191f3b20-istiod-ca-cert\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.853293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.853267 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-envoy\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.853580 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.853562 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-podinfo\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.868017 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.867996 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-token\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.868151 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.868134 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vg7qt\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-kube-api-access-vg7qt\") pod \"e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:43.890125 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:43.890101 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:44.036600 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.036576 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6"] Jul 06 16:55:44.038532 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:55:44.038500 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda41d0b_8915_4449_bcb4_549a191f3b20.slice/crio-008608a1d28e2338e446986ae40bb3b27eddb6bf827f367695863ddf9af30d20 WatchSource:0}: Error finding container 008608a1d28e2338e446986ae40bb3b27eddb6bf827f367695863ddf9af30d20: Status 404 returned error can't find the container with id 008608a1d28e2338e446986ae40bb3b27eddb6bf827f367695863ddf9af30d20 Jul 06 16:55:44.040730 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.040700 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:55:44.040804 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.040762 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:55:44.040850 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.040806 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:55:44.967108 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.967073 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" event={"ID":"bda41d0b-8915-4449-bcb4-549a191f3b20","Type":"ContainerStarted","Data":"11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f"} Jul 06 16:55:44.967528 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.967121 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" event={"ID":"bda41d0b-8915-4449-bcb4-549a191f3b20","Type":"ContainerStarted","Data":"008608a1d28e2338e446986ae40bb3b27eddb6bf827f367695863ddf9af30d20"} Jul 06 16:55:44.993645 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:44.993595 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podStartSLOduration=1.993579051 podStartE2EDuration="1.993579051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:55:44.991498368 +0000 UTC m=+1927.706717365" watchObservedRunningTime="2026-07-06 16:55:44.993579051 +0000 UTC m=+1927.708798025" Jul 06 16:55:45.891166 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:45.891131 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:45.895840 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:45.895804 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:45.972149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:45.972119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:45.973118 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:45.973099 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:55:52.323399 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.323367 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks"] Jul 06 16:55:52.323900 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.323868 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" Jul 06 16:55:52.323900 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.323883 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2937772-32a1-41a7-82be-2aa60e232732" containerName="istio-proxy" Jul 06 16:55:52.327534 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.327512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.347242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.347218 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-e8d6d5-openshift-default-dockercfg-rrhhg\"" Jul 06 16:55:52.375285 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.375251 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks"] Jul 06 16:55:52.429423 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429379 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-socket\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429604 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-credential-socket\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429604 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429484 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-certs\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429604 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429529 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-token\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429604 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-envoy\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429826 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429643 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-podinfo\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429826 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429681 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-data\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429826 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429745 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istiod-ca-cert\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.429826 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.429776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvz6r\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-kube-api-access-vvz6r\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530370 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-socket\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530370 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530380 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-credential-socket\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530400 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-certs\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530416 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-token\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-envoy\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530501 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-podinfo\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530532 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-data\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530562 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istiod-ca-cert\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.530616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.530598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvz6r\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-kube-api-access-vvz6r\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.531148 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.531118 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-certs\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.531319 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.531164 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-credential-socket\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.531319 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.531244 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-socket\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.531702 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.531632 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-data\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.532133 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.532108 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istiod-ca-cert\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.534465 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.534435 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-podinfo\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.540080 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.540059 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-envoy\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.545180 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.545159 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-token\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.563697 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.563672 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvz6r\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-kube-api-access-vvz6r\") pod \"e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.638172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.638118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:52.815188 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.815155 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks"] Jul 06 16:55:52.823893 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:55:52.823868 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ea7b78_c09b_4676_8e5a_2a4fec000f77.slice/crio-4512dc2bf4595b9be530d56c56230ce74ac1e89d1a487b2b59df722116ff256f WatchSource:0}: Error finding container 4512dc2bf4595b9be530d56c56230ce74ac1e89d1a487b2b59df722116ff256f: Status 404 returned error can't find the container with id 4512dc2bf4595b9be530d56c56230ce74ac1e89d1a487b2b59df722116ff256f Jul 06 16:55:52.825948 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.825910 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:55:52.826044 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.825993 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:55:52.826107 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:52.826039 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:55:53.009293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:53.009254 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" event={"ID":"25ea7b78-c09b-4676-8e5a-2a4fec000f77","Type":"ContainerStarted","Data":"6739fd4c75ad9a99cdd114700141294f9e6166fbd252ea989478d0e0e6d0c5ab"} Jul 06 16:55:53.009461 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:53.009310 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" event={"ID":"25ea7b78-c09b-4676-8e5a-2a4fec000f77","Type":"ContainerStarted","Data":"4512dc2bf4595b9be530d56c56230ce74ac1e89d1a487b2b59df722116ff256f"} Jul 06 16:55:53.051439 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:53.051389 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podStartSLOduration=1.051372035 podStartE2EDuration="1.051372035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:55:53.048775372 +0000 UTC m=+1935.763994349" watchObservedRunningTime="2026-07-06 16:55:53.051372035 +0000 UTC m=+1935.766591010" Jul 06 16:55:53.638749 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:53.638710 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:53.643773 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:53.643744 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:54.013815 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:54.013716 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:55:54.014727 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:55:54.014711 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:56:02.014965 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:02.014922 2563 patch_prober.go:28] interesting pod/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 16:56:02.015441 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:02.014991 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 16:56:04.064926 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:04.064893 2563 generic.go:356] "Generic (PLEG): container finished" podID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerID="11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f" exitCode=0 Jul 06 16:56:04.065384 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:04.064959 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" event={"ID":"bda41d0b-8915-4449-bcb4-549a191f3b20","Type":"ContainerDied","Data":"11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f"} Jul 06 16:56:04.065384 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:04.065327 2563 scope.go:117] "RemoveContainer" containerID="11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f" Jul 06 16:56:04.066340 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:04.066309 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:56:04.066449 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:04.066392 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:56:04.066449 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:04.066438 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:56:05.073968 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:05.073934 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" event={"ID":"bda41d0b-8915-4449-bcb4-549a191f3b20","Type":"ContainerStarted","Data":"5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398"} Jul 06 16:56:05.891261 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:05.891226 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:56:05.891620 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:05.891527 2563 patch_prober.go:28] interesting pod/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 16:56:05.891847 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:05.891815 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 16:56:06.891112 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:06.891065 2563 patch_prober.go:28] interesting pod/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 16:56:06.891592 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:06.891151 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 16:56:07.891483 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:07.891443 2563 patch_prober.go:28] interesting pod/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 16:56:07.892118 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:07.892085 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 16:56:08.891302 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:08.891257 2563 patch_prober.go:28] interesting pod/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 06 16:56:08.891660 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:08.891324 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 06 16:56:09.393080 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:09.392486 2563 patch_prober.go:28] interesting pod/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:33372->10.133.0.53:15021: read: connection reset by peer" start-of-body= Jul 06 16:56:09.393080 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:09.392557 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:33372->10.133.0.53:15021: read: connection reset by peer" Jul 06 16:56:09.895648 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:09.895620 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:56:09.896087 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:09.895911 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:56:09.896877 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:09.896854 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:56:10.106302 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:10.106266 2563 generic.go:356] "Generic (PLEG): container finished" podID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerID="6739fd4c75ad9a99cdd114700141294f9e6166fbd252ea989478d0e0e6d0c5ab" exitCode=0 Jul 06 16:56:10.106498 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:10.106345 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" event={"ID":"25ea7b78-c09b-4676-8e5a-2a4fec000f77","Type":"ContainerDied","Data":"6739fd4c75ad9a99cdd114700141294f9e6166fbd252ea989478d0e0e6d0c5ab"} Jul 06 16:56:10.106849 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:10.106820 2563 scope.go:117] "RemoveContainer" containerID="6739fd4c75ad9a99cdd114700141294f9e6166fbd252ea989478d0e0e6d0c5ab" Jul 06 16:56:10.107781 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:10.107756 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:56:10.107876 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:10.107860 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:56:10.107918 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:10.107895 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:56:11.122965 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:11.122915 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" event={"ID":"25ea7b78-c09b-4676-8e5a-2a4fec000f77","Type":"ContainerStarted","Data":"5928351d9173e10e6f47ecce2d2aadf224d241d6b6a6c0ed794f5254f7f3358b"} Jul 06 16:56:11.638785 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:11.638750 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:56:11.639040 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:11.639018 2563 patch_prober.go:28] interesting pod/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 16:56:11.639099 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:11.639079 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 16:56:12.638521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:12.638484 2563 patch_prober.go:28] interesting pod/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 16:56:12.638960 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:12.638541 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 16:56:13.639569 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:13.639537 2563 patch_prober.go:28] interesting pod/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 16:56:13.640030 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:13.639920 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 16:56:14.639473 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:14.639432 2563 patch_prober.go:28] interesting pod/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body= Jul 06 16:56:14.639684 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:14.639548 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" Jul 06 16:56:15.642875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:15.642844 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:56:15.643274 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:15.643134 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:56:15.643893 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:56:15.643878 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:57:53.385715 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:53.385682 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6"] Jul 06 16:57:53.386271 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:53.385926 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" containerID="cri-o://5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398" gracePeriod=30 Jul 06 16:57:54.897753 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:54.897720 2563 patch_prober.go:28] interesting pod/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 06 16:57:54.898210 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:54.897808 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 06 16:57:58.639222 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.639198 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:57:58.651212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.651130 2563 generic.go:356] "Generic (PLEG): container finished" podID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerID="5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398" exitCode=0 Jul 06 16:57:58.651212 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.651181 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" Jul 06 16:57:58.651417 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.651218 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" event={"ID":"bda41d0b-8915-4449-bcb4-549a191f3b20","Type":"ContainerDied","Data":"5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398"} Jul 06 16:57:58.651417 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.651314 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6" event={"ID":"bda41d0b-8915-4449-bcb4-549a191f3b20","Type":"ContainerDied","Data":"008608a1d28e2338e446986ae40bb3b27eddb6bf827f367695863ddf9af30d20"} Jul 06 16:57:58.651417 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.651345 2563 scope.go:117] "RemoveContainer" containerID="5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398" Jul 06 16:57:58.660158 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.660137 2563 scope.go:117] "RemoveContainer" containerID="11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f" Jul 06 16:57:58.670543 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.670481 2563 scope.go:117] "RemoveContainer" containerID="5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398" Jul 06 16:57:58.670907 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:57:58.670881 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398\": container with ID starting with 5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398 not found: ID does not exist" containerID="5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398" Jul 06 16:57:58.671076 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.671038 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398"} err="failed to get container status \"5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398\": rpc error: code = NotFound desc = could not find container \"5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398\": container with ID starting with 5416519836f667d2029e0aadf5ec87baaf25a9de87e8952f3532e6e70be88398 not found: ID does not exist" Jul 06 16:57:58.671187 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.671080 2563 scope.go:117] "RemoveContainer" containerID="11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f" Jul 06 16:57:58.671396 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:57:58.671375 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f\": container with ID starting with 11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f not found: ID does not exist" containerID="11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f" Jul 06 16:57:58.671531 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.671481 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f"} err="failed to get container status \"11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f\": rpc error: code = NotFound desc = could not find container \"11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f\": container with ID starting with 11078c09880ac78ef783c227b6e6e1dc9754307d3fd0c8c9cab92d16201baf5f not found: ID does not exist" Jul 06 16:57:58.732088 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732057 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-socket\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732100 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vg7qt\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-kube-api-access-vg7qt\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732127 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-data\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732168 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-certs\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732193 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-podinfo\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732252 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-credential-socket\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732283 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda41d0b-8915-4449-bcb4-549a191f3b20-istiod-ca-cert\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732308 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-token\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732362 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732333 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-envoy\") pod \"bda41d0b-8915-4449-bcb4-549a191f3b20\" (UID: \"bda41d0b-8915-4449-bcb4-549a191f3b20\") " Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732402 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732412 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732550 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732683 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732692 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda41d0b-8915-4449-bcb4-549a191f3b20-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732701 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.732879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.732733 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.733257 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.733182 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-data" (OuterVolumeSpecName: "istio-data") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:57:58.734412 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.734384 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-kube-api-access-vg7qt" (OuterVolumeSpecName: "kube-api-access-vg7qt") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "kube-api-access-vg7qt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:57:58.734508 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.734434 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:57:58.734547 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.734521 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-token" (OuterVolumeSpecName: "istio-token") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:57:58.734611 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.734598 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bda41d0b-8915-4449-bcb4-549a191f3b20" (UID: "bda41d0b-8915-4449-bcb4-549a191f3b20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:57:58.833955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.833900 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.833955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.833935 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda41d0b-8915-4449-bcb4-549a191f3b20-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.833955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.833950 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.833955 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.833963 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.834269 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.833976 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vg7qt\" (UniqueName: \"kubernetes.io/projected/bda41d0b-8915-4449-bcb4-549a191f3b20-kube-api-access-vg7qt\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.834269 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.833989 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda41d0b-8915-4449-bcb4-549a191f3b20-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:57:58.978739 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.978702 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6"] Jul 06 16:57:58.985470 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:58.985442 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-da00b0-openshift-default-7686bffd5f-79cg6"] Jul 06 16:57:59.793182 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:57:59.793148 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" path="/var/lib/kubelet/pods/bda41d0b-8915-4449-bcb4-549a191f3b20/volumes" Jul 06 16:58:01.558849 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:01.558751 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks"] Jul 06 16:58:01.559303 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:01.558987 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" containerID="cri-o://5928351d9173e10e6f47ecce2d2aadf224d241d6b6a6c0ed794f5254f7f3358b" gracePeriod=30 Jul 06 16:58:02.637889 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.637858 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft"] Jul 06 16:58:02.638275 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.638258 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" Jul 06 16:58:02.638275 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.638269 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" Jul 06 16:58:02.638381 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.638368 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" Jul 06 16:58:02.638466 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.638382 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" Jul 06 16:58:02.642959 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.642939 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.648049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.648025 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0064ec-openshift-default-dockercfg-gbwhh\"" Jul 06 16:58:02.680379 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.680350 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft"] Jul 06 16:58:02.770292 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770250 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253f22e1-7510-4adf-854c-64d4304b5bd9-istiod-ca-cert\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770466 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-data\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770466 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770366 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7tls\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-kube-api-access-j7tls\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770466 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770386 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-certs\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770466 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770435 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-credential-socket\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770608 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770468 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-podinfo\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770608 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770494 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-token\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770608 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770529 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-socket\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.770608 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.770544 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-envoy\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.871765 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.871727 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-socket\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.871765 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.871767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-envoy\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.871853 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253f22e1-7510-4adf-854c-64d4304b5bd9-istiod-ca-cert\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.871927 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-data\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.871971 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7tls\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-kube-api-access-j7tls\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.871996 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-certs\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872049 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872021 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-credential-socket\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872065 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-podinfo\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-token\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872341 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872205 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-socket\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872503 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872476 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-data\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872556 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872498 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-certs\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872629 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872592 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253f22e1-7510-4adf-854c-64d4304b5bd9-istiod-ca-cert\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.872666 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.872646 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-credential-socket\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.874113 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.874092 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-envoy\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.874406 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.874387 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-podinfo\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.888432 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.888353 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-token\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.888556 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.888469 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7tls\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-kube-api-access-j7tls\") pod \"e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:02.953881 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:02.953844 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:03.121848 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:58:03.121821 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod253f22e1_7510_4adf_854c_64d4304b5bd9.slice/crio-7cc7bc3955e3c2d0b92bec09d8f45000c0647a994ff42c9d7dec232ee17bae0a WatchSource:0}: Error finding container 7cc7bc3955e3c2d0b92bec09d8f45000c0647a994ff42c9d7dec232ee17bae0a: Status 404 returned error can't find the container with id 7cc7bc3955e3c2d0b92bec09d8f45000c0647a994ff42c9d7dec232ee17bae0a Jul 06 16:58:03.123675 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.123637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft"] Jul 06 16:58:03.124360 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.124327 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:03.124441 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.124394 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:03.124441 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.124424 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:03.679488 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.679455 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" event={"ID":"253f22e1-7510-4adf-854c-64d4304b5bd9","Type":"ContainerStarted","Data":"ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74"} Jul 06 16:58:03.679922 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.679510 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" event={"ID":"253f22e1-7510-4adf-854c-64d4304b5bd9","Type":"ContainerStarted","Data":"7cc7bc3955e3c2d0b92bec09d8f45000c0647a994ff42c9d7dec232ee17bae0a"} Jul 06 16:58:03.704443 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.704387 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podStartSLOduration=1.704370005 podStartE2EDuration="1.704370005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:03.703086156 +0000 UTC m=+2066.418305188" watchObservedRunningTime="2026-07-06 16:58:03.704370005 +0000 UTC m=+2066.419588978" Jul 06 16:58:03.954529 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:03.954444 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:04.959066 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:04.959038 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:05.690368 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:05.690333 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:05.691356 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:05.691338 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:06.698683 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.698654 2563 generic.go:356] "Generic (PLEG): container finished" podID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerID="5928351d9173e10e6f47ecce2d2aadf224d241d6b6a6c0ed794f5254f7f3358b" exitCode=0 Jul 06 16:58:06.699063 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.698727 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" event={"ID":"25ea7b78-c09b-4676-8e5a-2a4fec000f77","Type":"ContainerDied","Data":"5928351d9173e10e6f47ecce2d2aadf224d241d6b6a6c0ed794f5254f7f3358b"} Jul 06 16:58:06.699063 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.698784 2563 scope.go:117] "RemoveContainer" containerID="6739fd4c75ad9a99cdd114700141294f9e6166fbd252ea989478d0e0e6d0c5ab" Jul 06 16:58:06.815744 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.815722 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:58:06.910717 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910681 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-certs\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.910717 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910717 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-envoy\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910743 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvz6r\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-kube-api-access-vvz6r\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910824 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-socket\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910851 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-data\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910877 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-credential-socket\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910906 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istiod-ca-cert\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910932 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-token\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911020 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.910994 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-podinfo\") pod \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\" (UID: \"25ea7b78-c09b-4676-8e5a-2a4fec000f77\") " Jul 06 16:58:06.911359 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911092 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:06.911359 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911183 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:06.911359 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911330 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:06.911359 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911350 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:06.911556 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911438 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:58:06.911556 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911513 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:06.911716 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.911696 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-data" (OuterVolumeSpecName: "istio-data") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:06.913504 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.913468 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-token" (OuterVolumeSpecName: "istio-token") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:58:06.913614 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.913512 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:06.913614 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.913515 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:58:06.913614 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:06.913536 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-kube-api-access-vvz6r" (OuterVolumeSpecName: "kube-api-access-vvz6r") pod "25ea7b78-c09b-4676-8e5a-2a4fec000f77" (UID: "25ea7b78-c09b-4676-8e5a-2a4fec000f77"). InnerVolumeSpecName "kube-api-access-vvz6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:58:07.012518 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012484 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.012518 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012513 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.012518 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012526 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.012761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012534 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.012761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012542 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.012761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012551 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvz6r\" (UniqueName: \"kubernetes.io/projected/25ea7b78-c09b-4676-8e5a-2a4fec000f77-kube-api-access-vvz6r\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.012761 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.012559 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25ea7b78-c09b-4676-8e5a-2a4fec000f77-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:07.705865 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.705827 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" event={"ID":"25ea7b78-c09b-4676-8e5a-2a4fec000f77","Type":"ContainerDied","Data":"4512dc2bf4595b9be530d56c56230ce74ac1e89d1a487b2b59df722116ff256f"} Jul 06 16:58:07.705865 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.705851 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks" Jul 06 16:58:07.706419 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.705902 2563 scope.go:117] "RemoveContainer" containerID="5928351d9173e10e6f47ecce2d2aadf224d241d6b6a6c0ed794f5254f7f3358b" Jul 06 16:58:07.731870 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.731840 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks"] Jul 06 16:58:07.738608 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.738575 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-e8d6d5-openshift-default-59b977597f-ncrks"] Jul 06 16:58:07.799253 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:07.799220 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" path="/var/lib/kubelet/pods/25ea7b78-c09b-4676-8e5a-2a4fec000f77/volumes" Jul 06 16:58:11.066307 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.066275 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs"] Jul 06 16:58:11.066760 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.066725 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" Jul 06 16:58:11.066760 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.066740 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" Jul 06 16:58:11.066856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.066765 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" Jul 06 16:58:11.066856 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.066771 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda41d0b-8915-4449-bcb4-549a191f3b20" containerName="istio-proxy" Jul 06 16:58:11.066924 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.066856 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" Jul 06 16:58:11.071243 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.071224 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.078714 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.078691 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7ed8f1-openshift-default-dockercfg-lnhhb\"" Jul 06 16:58:11.090428 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.090393 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs"] Jul 06 16:58:11.152992 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.152947 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00fa166-a274-47d6-bb43-3b4b4f4627be-istiod-ca-cert\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.152992 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.152993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-certs\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153012 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-podinfo\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-socket\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153135 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-token\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153182 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-credential-socket\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153210 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-data\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153268 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153227 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwq4n\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-kube-api-access-hwq4n\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.153596 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.153360 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-envoy\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.254744 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254707 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-socket\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.254744 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254742 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-token\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-credential-socket\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254781 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-data\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254836 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwq4n\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-kube-api-access-hwq4n\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254909 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-envoy\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254943 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00fa166-a274-47d6-bb43-3b4b4f4627be-istiod-ca-cert\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.254979 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-certs\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255054 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.255000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-podinfo\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255488 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.255234 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-socket\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255488 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.255285 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-data\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255488 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.255315 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-credential-socket\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255488 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.255438 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-certs\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.255758 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.255734 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00fa166-a274-47d6-bb43-3b4b4f4627be-istiod-ca-cert\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.257468 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.257449 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-envoy\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.257573 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.257538 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-podinfo\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.267497 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.267469 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwq4n\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-kube-api-access-hwq4n\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.268751 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.268721 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-token\") pod \"e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.381740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.381687 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:11.602925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.602898 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs"] Jul 06 16:58:11.605691 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:58:11.605667 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc00fa166_a274_47d6_bb43_3b4b4f4627be.slice/crio-b6091a4fe92e141849c5b53be0b1183cf07335aa804179e43f01d376c35a871b WatchSource:0}: Error finding container b6091a4fe92e141849c5b53be0b1183cf07335aa804179e43f01d376c35a871b: Status 404 returned error can't find the container with id b6091a4fe92e141849c5b53be0b1183cf07335aa804179e43f01d376c35a871b Jul 06 16:58:11.607876 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.607838 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:11.607942 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.607908 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:11.607942 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.607936 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:11.734411 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.734374 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" event={"ID":"c00fa166-a274-47d6-bb43-3b4b4f4627be","Type":"ContainerStarted","Data":"7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a"} Jul 06 16:58:11.734631 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.734448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" event={"ID":"c00fa166-a274-47d6-bb43-3b4b4f4627be","Type":"ContainerStarted","Data":"b6091a4fe92e141849c5b53be0b1183cf07335aa804179e43f01d376c35a871b"} Jul 06 16:58:11.775740 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:11.775689 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podStartSLOduration=0.775673347 podStartE2EDuration="775.673347ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:11.775413745 +0000 UTC m=+2074.490632732" watchObservedRunningTime="2026-07-06 16:58:11.775673347 +0000 UTC m=+2074.490892320" Jul 06 16:58:12.382346 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:12.382302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:12.388748 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:12.388721 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:12.739325 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:12.739238 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:12.740149 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:12.740128 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:21.731084 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.730091 2563 patch_prober.go:28] interesting pod/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 06 16:58:21.731084 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.730187 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 06 16:58:21.788106 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.788075 2563 generic.go:356] "Generic (PLEG): container finished" podID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerID="ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74" exitCode=0 Jul 06 16:58:21.795323 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.795273 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" event={"ID":"253f22e1-7510-4adf-854c-64d4304b5bd9","Type":"ContainerDied","Data":"ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74"} Jul 06 16:58:21.795920 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.795769 2563 scope.go:117] "RemoveContainer" containerID="ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74" Jul 06 16:58:21.798435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.797571 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:21.798435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.797670 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:21.798435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:21.797729 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:22.798508 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:22.798474 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" event={"ID":"253f22e1-7510-4adf-854c-64d4304b5bd9","Type":"ContainerStarted","Data":"edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2"} Jul 06 16:58:22.954747 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:22.954710 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:22.955032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:22.955006 2563 patch_prober.go:28] interesting pod/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 16:58:22.955118 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:22.955078 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 16:58:23.954698 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:23.954665 2563 patch_prober.go:28] interesting pod/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 16:58:23.955167 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:23.954717 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 16:58:24.954502 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:24.954461 2563 patch_prober.go:28] interesting pod/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 16:58:24.954693 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:24.954518 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 16:58:25.954517 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:25.954430 2563 patch_prober.go:28] interesting pod/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 06 16:58:25.954875 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:25.954515 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 06 16:58:26.958152 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:26.958120 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:26.958536 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:26.958513 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:26.959321 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:26.959302 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:28.623086 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.622358 2563 patch_prober.go:28] interesting pod/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:34050->10.133.0.55:15021: read: connection reset by peer" start-of-body= Jul 06 16:58:28.623086 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.622437 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": read tcp 10.133.0.2:34050->10.133.0.55:15021: read: connection reset by peer" Jul 06 16:58:28.838616 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.838586 2563 generic.go:356] "Generic (PLEG): container finished" podID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerID="7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a" exitCode=0 Jul 06 16:58:28.838772 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.838661 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" event={"ID":"c00fa166-a274-47d6-bb43-3b4b4f4627be","Type":"ContainerDied","Data":"7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a"} Jul 06 16:58:28.839192 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.839177 2563 scope.go:117] "RemoveContainer" containerID="7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a" Jul 06 16:58:28.840141 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.840115 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:28.840201 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.840179 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:28.840238 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:28.840210 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:29.844894 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:29.844858 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" event={"ID":"c00fa166-a274-47d6-bb43-3b4b4f4627be","Type":"ContainerStarted","Data":"f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5"} Jul 06 16:58:30.381879 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:30.381838 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:30.382062 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:30.382034 2563 patch_prober.go:28] interesting pod/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 16:58:30.382111 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:30.382084 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 16:58:31.382580 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:31.382549 2563 patch_prober.go:28] interesting pod/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 16:58:31.383082 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:31.382600 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 16:58:32.382066 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:32.382031 2563 patch_prober.go:28] interesting pod/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 16:58:32.382257 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:32.382082 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 16:58:33.382068 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:33.382035 2563 patch_prober.go:28] interesting pod/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" start-of-body= Jul 06 16:58:33.382531 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:33.382086 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.55:15021/healthz/ready\": dial tcp 10.133.0.55:15021: connect: connection refused" Jul 06 16:58:33.562637 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:33.562603 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft"] Jul 06 16:58:33.562846 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:33.562809 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" containerID="cri-o://edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2" gracePeriod=30 Jul 06 16:58:34.385508 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:34.385481 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:34.385945 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:34.385751 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:34.386306 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:34.386289 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:37.876216 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:37.876182 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:58:37.882188 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:37.882155 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 16:58:38.817234 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.817213 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:38.891734 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.891702 2563 generic.go:356] "Generic (PLEG): container finished" podID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerID="edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2" exitCode=0 Jul 06 16:58:38.892215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.891767 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" Jul 06 16:58:38.892215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.891778 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" event={"ID":"253f22e1-7510-4adf-854c-64d4304b5bd9","Type":"ContainerDied","Data":"edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2"} Jul 06 16:58:38.892215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.891853 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft" event={"ID":"253f22e1-7510-4adf-854c-64d4304b5bd9","Type":"ContainerDied","Data":"7cc7bc3955e3c2d0b92bec09d8f45000c0647a994ff42c9d7dec232ee17bae0a"} Jul 06 16:58:38.892215 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.891886 2563 scope.go:117] "RemoveContainer" containerID="edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2" Jul 06 16:58:38.900565 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.900550 2563 scope.go:117] "RemoveContainer" containerID="ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74" Jul 06 16:58:38.909431 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.909412 2563 scope.go:117] "RemoveContainer" containerID="edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2" Jul 06 16:58:38.909689 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:58:38.909670 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2\": container with ID starting with edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2 not found: ID does not exist" containerID="edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2" Jul 06 16:58:38.909763 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.909704 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2"} err="failed to get container status \"edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2\": rpc error: code = NotFound desc = could not find container \"edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2\": container with ID starting with edc30a8bd43f6dc52e8d936033e8bab306eb52d98e7ec9f3b3901f627be834e2 not found: ID does not exist" Jul 06 16:58:38.909763 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.909741 2563 scope.go:117] "RemoveContainer" containerID="ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74" Jul 06 16:58:38.910021 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:58:38.910004 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74\": container with ID starting with ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74 not found: ID does not exist" containerID="ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74" Jul 06 16:58:38.910068 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.910026 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74"} err="failed to get container status \"ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74\": rpc error: code = NotFound desc = could not find container \"ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74\": container with ID starting with ffea1777a7d3af566331ba92dee5e2e2b2a5c064cfd95946d9bd37e9c4132e74 not found: ID does not exist" Jul 06 16:58:38.943809 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.943279 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-podinfo\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.943959 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.943857 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7tls\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-kube-api-access-j7tls\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.945717 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.944278 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253f22e1-7510-4adf-854c-64d4304b5bd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:58:38.946230 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946182 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.943924 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253f22e1-7510-4adf-854c-64d4304b5bd9-istiod-ca-cert\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946403 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-certs\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946465 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-credential-socket\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946494 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-envoy\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946531 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-socket\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-token\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946600 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-data\") pod \"253f22e1-7510-4adf-854c-64d4304b5bd9\" (UID: \"253f22e1-7510-4adf-854c-64d4304b5bd9\") " Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.946714 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-kube-api-access-j7tls" (OuterVolumeSpecName: "kube-api-access-j7tls") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "kube-api-access-j7tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.947005 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.947022 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7tls\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-kube-api-access-j7tls\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.947036 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253f22e1-7510-4adf-854c-64d4304b5bd9-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.947766 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:38.948242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.948006 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:38.948979 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.948955 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:38.949188 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.949161 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:38.949351 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.949332 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:38.952255 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:38.952227 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "253f22e1-7510-4adf-854c-64d4304b5bd9" (UID: "253f22e1-7510-4adf-854c-64d4304b5bd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:58:39.047978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.047929 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:39.047978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.047973 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:39.047978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.047984 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:39.047978 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.047992 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:39.048242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.048000 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:39.048242 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.048008 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253f22e1-7510-4adf-854c-64d4304b5bd9-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:39.222272 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.222235 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft"] Jul 06 16:58:39.226896 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.226872 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0064ec-openshift-default-6cb7547c7f-tf6ft"] Jul 06 16:58:39.792913 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:39.792882 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" path="/var/lib/kubelet/pods/253f22e1-7510-4adf-854c-64d4304b5bd9/volumes" Jul 06 16:58:40.878851 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:40.878819 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs"] Jul 06 16:58:40.879244 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:40.878998 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" containerID="cri-o://f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5" gracePeriod=30 Jul 06 16:58:41.757861 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.757822 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj"] Jul 06 16:58:41.758304 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758286 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" Jul 06 16:58:41.758363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758306 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" Jul 06 16:58:41.758363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758332 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" Jul 06 16:58:41.758363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758339 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" Jul 06 16:58:41.758363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758347 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" Jul 06 16:58:41.758363 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758352 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" Jul 06 16:58:41.758536 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758414 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ea7b78-c09b-4676-8e5a-2a4fec000f77" containerName="istio-proxy" Jul 06 16:58:41.758536 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758422 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" Jul 06 16:58:41.758536 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.758429 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="253f22e1-7510-4adf-854c-64d4304b5bd9" containerName="istio-proxy" Jul 06 16:58:41.766882 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.766859 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.769914 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.769895 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-0e87f3-openshift-default-dockercfg-hpn9t\"" Jul 06 16:58:41.774957 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.774933 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj"] Jul 06 16:58:41.873600 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-data\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.873776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873614 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-token\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.873776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-certs\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.873776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873733 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-credential-socket\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.873776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-envoy\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.873776 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873776 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-socket\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.874043 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hwwt\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-kube-api-access-6hwwt\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.874043 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873916 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-podinfo\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.874043 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.873949 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istiod-ca-cert\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.975442 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975405 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-podinfo\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.975442 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975443 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istiod-ca-cert\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975504 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-data\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975539 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-token\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975572 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-certs\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975627 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-credential-socket\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975649 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-envoy\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975682 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-socket\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975709 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hwwt\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-kube-api-access-6hwwt\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976032 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.975986 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-data\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976350 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.976029 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-certs\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976350 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.976118 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-credential-socket\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976350 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.976249 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-socket\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.976350 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.976312 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istiod-ca-cert\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.977954 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.977929 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-envoy\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.978059 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.977966 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-podinfo\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.985995 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.985959 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-token\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:41.986351 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:41.986091 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hwwt\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-kube-api-access-6hwwt\") pod \"e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:42.079894 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.079766 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:42.242261 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.242239 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj"] Jul 06 16:58:42.244541 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:58:42.244516 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c7de4a6_9f91_44f3_84fc_7a5793fc157d.slice/crio-718792046d422bcde5d84bdc3854e076ebad11f8a2ed19d985feef8b5eb0b4db WatchSource:0}: Error finding container 718792046d422bcde5d84bdc3854e076ebad11f8a2ed19d985feef8b5eb0b4db: Status 404 returned error can't find the container with id 718792046d422bcde5d84bdc3854e076ebad11f8a2ed19d985feef8b5eb0b4db Jul 06 16:58:42.246726 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.246696 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:42.246811 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.246763 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:42.246863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.246835 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:42.915447 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.915409 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" event={"ID":"6c7de4a6-9f91-44f3-84fc-7a5793fc157d","Type":"ContainerStarted","Data":"80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a"} Jul 06 16:58:42.915659 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.915471 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" event={"ID":"6c7de4a6-9f91-44f3-84fc-7a5793fc157d","Type":"ContainerStarted","Data":"718792046d422bcde5d84bdc3854e076ebad11f8a2ed19d985feef8b5eb0b4db"} Jul 06 16:58:42.943691 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:42.943642 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podStartSLOduration=1.943625586 podStartE2EDuration="1.943625586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:42.942183552 +0000 UTC m=+2105.657402570" watchObservedRunningTime="2026-07-06 16:58:42.943625586 +0000 UTC m=+2105.658844560" Jul 06 16:58:43.080435 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:43.080405 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:43.085180 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:43.085159 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:43.920847 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:43.920818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:43.921766 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:43.921747 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:58:46.140186 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.140128 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:46.214277 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214234 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-data\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214277 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214282 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hwq4n\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-kube-api-access-hwq4n\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214389 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-token\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214426 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00fa166-a274-47d6-bb43-3b4b4f4627be-istiod-ca-cert\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214454 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-certs\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214483 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-socket\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214521 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214514 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-credential-socket\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214763 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214538 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-envoy\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214763 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214589 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-podinfo\") pod \"c00fa166-a274-47d6-bb43-3b4b4f4627be\" (UID: \"c00fa166-a274-47d6-bb43-3b4b4f4627be\") " Jul 06 16:58:46.214972 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.214921 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:46.215037 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.215018 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.215127 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.215109 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:46.215269 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.215237 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-data" (OuterVolumeSpecName: "istio-data") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:46.215394 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.215310 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c00fa166-a274-47d6-bb43-3b4b4f4627be-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 16:58:46.215394 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.215326 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:46.216652 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.216629 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-kube-api-access-hwq4n" (OuterVolumeSpecName: "kube-api-access-hwq4n") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "kube-api-access-hwq4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:58:46.216743 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.216667 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-token" (OuterVolumeSpecName: "istio-token") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 16:58:46.216787 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.216775 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 16:58:46.216968 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.216945 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c00fa166-a274-47d6-bb43-3b4b4f4627be" (UID: "c00fa166-a274-47d6-bb43-3b4b4f4627be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 16:58:46.315876 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315845 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.315876 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315871 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c00fa166-a274-47d6-bb43-3b4b4f4627be-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.315876 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315880 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.316085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315889 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.316085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315898 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.316085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315906 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.316085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315914 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c00fa166-a274-47d6-bb43-3b4b4f4627be-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.316085 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.315922 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwq4n\" (UniqueName: \"kubernetes.io/projected/c00fa166-a274-47d6-bb43-3b4b4f4627be-kube-api-access-hwq4n\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 16:58:46.938293 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.938257 2563 generic.go:356] "Generic (PLEG): container finished" podID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerID="f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5" exitCode=0 Jul 06 16:58:46.938469 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.938316 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" Jul 06 16:58:46.938469 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.938312 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" event={"ID":"c00fa166-a274-47d6-bb43-3b4b4f4627be","Type":"ContainerDied","Data":"f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5"} Jul 06 16:58:46.938469 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.938452 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs" event={"ID":"c00fa166-a274-47d6-bb43-3b4b4f4627be","Type":"ContainerDied","Data":"b6091a4fe92e141849c5b53be0b1183cf07335aa804179e43f01d376c35a871b"} Jul 06 16:58:46.938574 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.938484 2563 scope.go:117] "RemoveContainer" containerID="f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5" Jul 06 16:58:46.948024 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.948007 2563 scope.go:117] "RemoveContainer" containerID="7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a" Jul 06 16:58:46.958280 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.958257 2563 scope.go:117] "RemoveContainer" containerID="f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5" Jul 06 16:58:46.958561 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:58:46.958543 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5\": container with ID starting with f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5 not found: ID does not exist" containerID="f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5" Jul 06 16:58:46.958634 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.958568 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5"} err="failed to get container status \"f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5\": rpc error: code = NotFound desc = could not find container \"f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5\": container with ID starting with f2c010af5728b9a72d7f66ac9e6b4d01c3201d9c4e7e6aedbb41675cb1775df5 not found: ID does not exist" Jul 06 16:58:46.958634 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.958584 2563 scope.go:117] "RemoveContainer" containerID="7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a" Jul 06 16:58:46.958831 ip-10-0-136-202 kubenswrapper[2563]: E0706 16:58:46.958811 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a\": container with ID starting with 7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a not found: ID does not exist" containerID="7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a" Jul 06 16:58:46.958882 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.958836 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a"} err="failed to get container status \"7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a\": rpc error: code = NotFound desc = could not find container \"7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a\": container with ID starting with 7d4eb5f5c21f632632ef589f794f98e53377ea0a1855fd004acb22989a25f91a not found: ID does not exist" Jul 06 16:58:46.963431 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.963403 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs"] Jul 06 16:58:46.966305 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:46.966285 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7ed8f1-openshift-default-6fc7cf9495-prczs"] Jul 06 16:58:47.794155 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:47.794129 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" path="/var/lib/kubelet/pods/c00fa166-a274-47d6-bb43-3b4b4f4627be/volumes" Jul 06 16:58:50.039511 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.039474 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj"] Jul 06 16:58:50.040034 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.040006 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" Jul 06 16:58:50.040034 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.040021 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" Jul 06 16:58:50.040034 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.040031 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" Jul 06 16:58:50.040136 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.040037 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" Jul 06 16:58:50.040136 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.040121 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" Jul 06 16:58:50.040136 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.040130 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="c00fa166-a274-47d6-bb43-3b4b4f4627be" containerName="istio-proxy" Jul 06 16:58:50.044627 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.044605 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.047478 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.047461 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-327802-openshift-default-dockercfg-p79b4\"" Jul 06 16:58:50.055198 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.055173 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj"] Jul 06 16:58:50.148971 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.148934 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-socket\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.148982 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istiod-ca-cert\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6swq\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-kube-api-access-f6swq\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149172 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149153 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-certs\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149338 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-token\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149338 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149303 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-podinfo\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149444 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149343 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-credential-socket\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149444 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149408 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-envoy\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.149541 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.149441 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-data\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250128 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250089 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-socket\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250128 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istiod-ca-cert\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250385 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250181 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6swq\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-kube-api-access-f6swq\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250385 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250213 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-certs\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250385 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250247 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-token\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250385 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250300 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-podinfo\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250385 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-credential-socket\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250656 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250542 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-socket\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250656 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250594 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-certs\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250768 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250656 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-credential-socket\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250768 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250366 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-envoy\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250768 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250740 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-data\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.250958 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.250933 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istiod-ca-cert\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.251057 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.251037 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-data\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.252709 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.252685 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-envoy\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.252834 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.252765 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-podinfo\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.260455 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.260431 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-token\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.260578 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.260557 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6swq\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-kube-api-access-f6swq\") pod \"e2e-shared-b-327802-openshift-default-f989755c9-8qvfj\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.355680 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.355646 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:50.500330 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.500301 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj"] Jul 06 16:58:50.502129 ip-10-0-136-202 kubenswrapper[2563]: W0706 16:58:50.502098 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4024fa4d_7e3a_4079_ac5e_d3009cdbe2da.slice/crio-2a681ac7a2635189840f8c48f02dd4f726d8e593c754af13a34544bd97fdc94a WatchSource:0}: Error finding container 2a681ac7a2635189840f8c48f02dd4f726d8e593c754af13a34544bd97fdc94a: Status 404 returned error can't find the container with id 2a681ac7a2635189840f8c48f02dd4f726d8e593c754af13a34544bd97fdc94a Jul 06 16:58:50.504343 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.504295 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:50.504439 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.504393 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:50.504480 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.504443 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:50.963656 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.963624 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" event={"ID":"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da","Type":"ContainerStarted","Data":"bb318dc895d92952936f7240d0f2f8f6d780c7f96991e6060a47b216db62afdb"} Jul 06 16:58:50.963863 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:50.963673 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" event={"ID":"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da","Type":"ContainerStarted","Data":"2a681ac7a2635189840f8c48f02dd4f726d8e593c754af13a34544bd97fdc94a"} Jul 06 16:58:51.001282 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:51.001231 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" podStartSLOduration=1.001213274 podStartE2EDuration="1.001213274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 16:58:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 16:58:51.000313015 +0000 UTC m=+2113.715532014" watchObservedRunningTime="2026-07-06 16:58:51.001213274 +0000 UTC m=+2113.716432247" Jul 06 16:58:51.356257 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:51.356229 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:51.361146 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:51.361124 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:51.969851 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:51.969818 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:51.970676 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:51.970657 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:58:58.922980 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:58.922895 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:58:58.922980 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:58.922963 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:58:59.013326 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:59.013287 2563 generic.go:356] "Generic (PLEG): container finished" podID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerID="80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a" exitCode=0 Jul 06 16:58:59.013535 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:59.013364 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" event={"ID":"6c7de4a6-9f91-44f3-84fc-7a5793fc157d","Type":"ContainerDied","Data":"80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a"} Jul 06 16:58:59.013925 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:59.013902 2563 scope.go:117] "RemoveContainer" containerID="80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a" Jul 06 16:58:59.015350 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:59.015307 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:59.015447 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:59.015407 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:58:59.015507 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:58:59.015457 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:59:00.028644 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:00.028608 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" event={"ID":"6c7de4a6-9f91-44f3-84fc-7a5793fc157d","Type":"ContainerStarted","Data":"4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5"} Jul 06 16:59:00.079948 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:00.079918 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:59:00.080204 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:00.080184 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:59:00.080266 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:00.080238 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:59:01.080924 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:01.080891 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:59:01.081317 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:01.080941 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:59:02.080600 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:02.080558 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:59:02.080783 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:02.080623 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:59:03.080841 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:03.080804 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:59:03.081326 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:03.080868 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:59:04.081158 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:04.081119 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:59:04.081557 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:04.081178 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:59:05.080764 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:05.080735 2563 patch_prober.go:28] interesting pod/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 06 16:59:05.080967 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:05.080831 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 06 16:59:06.064733 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.064699 2563 generic.go:356] "Generic (PLEG): container finished" podID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerID="bb318dc895d92952936f7240d0f2f8f6d780c7f96991e6060a47b216db62afdb" exitCode=0 Jul 06 16:59:06.065180 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.064745 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" event={"ID":"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da","Type":"ContainerDied","Data":"bb318dc895d92952936f7240d0f2f8f6d780c7f96991e6060a47b216db62afdb"} Jul 06 16:59:06.065180 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.065170 2563 scope.go:117] "RemoveContainer" containerID="bb318dc895d92952936f7240d0f2f8f6d780c7f96991e6060a47b216db62afdb" Jul 06 16:59:06.066177 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.066143 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:59:06.066262 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.066227 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:59:06.066313 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.066266 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 06 16:59:06.085102 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.085082 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:59:06.085376 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.085346 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:59:06.086176 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.086160 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 16:59:06.970033 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:06.969992 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:59:07.072001 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:07.071956 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" event={"ID":"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da","Type":"ContainerStarted","Data":"51ca34744f7f10fb934bcd72f96f38e41f0dcb7c17fbfbc96ffe303a743e877a"} Jul 06 16:59:07.356491 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:07.356456 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:59:07.357837 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:07.357781 2563 patch_prober.go:28] interesting pod/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 16:59:07.357970 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:07.357870 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 16:59:08.356045 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:08.356011 2563 patch_prober.go:28] interesting pod/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 16:59:08.356482 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:08.356083 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 16:59:09.356707 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:09.356674 2563 patch_prober.go:28] interesting pod/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 16:59:09.357090 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:09.356739 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 16:59:10.356063 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:10.356023 2563 patch_prober.go:28] interesting pod/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 06 16:59:10.356240 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:10.356082 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 06 16:59:11.360567 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:11.360530 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:59:11.361116 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:11.361095 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 16:59:11.361825 ip-10-0-136-202 kubenswrapper[2563]: I0706 16:59:11.361783 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 17:03:37.909418 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:03:37.909376 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 17:03:37.917746 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:03:37.917725 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56nhz_5afa0111-b488-47f0-b10e-2c657d2f9057/ovn-acl-logging/0.log" Jul 06 17:05:04.486844 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.486810 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7036-140473332468848"] Jul 06 17:05:04.490432 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.490415 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:04.500958 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.500935 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-140473332468848"] Jul 06 17:05:04.622739 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.622704 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm92f\" (UniqueName: \"kubernetes.io/projected/ac602977-0938-43c7-8c38-8a511bb6f175-kube-api-access-hm92f\") pod \"test-curl-7036-140473332468848\" (UID: \"ac602977-0938-43c7-8c38-8a511bb6f175\") " pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:04.724023 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.723987 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm92f\" (UniqueName: \"kubernetes.io/projected/ac602977-0938-43c7-8c38-8a511bb6f175-kube-api-access-hm92f\") pod \"test-curl-7036-140473332468848\" (UID: \"ac602977-0938-43c7-8c38-8a511bb6f175\") " pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:04.735089 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.735064 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm92f\" (UniqueName: \"kubernetes.io/projected/ac602977-0938-43c7-8c38-8a511bb6f175-kube-api-access-hm92f\") pod \"test-curl-7036-140473332468848\" (UID: \"ac602977-0938-43c7-8c38-8a511bb6f175\") " pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:04.799380 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.799306 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:04.950874 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.950850 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7036-140473332468848"] Jul 06 17:05:04.953585 ip-10-0-136-202 kubenswrapper[2563]: W0706 17:05:04.953557 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac602977_0938_43c7_8c38_8a511bb6f175.slice/crio-039860fd15f707535af3d52ced7bce57dc12084b5179e8acc0f7b7b0feb1633e WatchSource:0}: Error finding container 039860fd15f707535af3d52ced7bce57dc12084b5179e8acc0f7b7b0feb1633e: Status 404 returned error can't find the container with id 039860fd15f707535af3d52ced7bce57dc12084b5179e8acc0f7b7b0feb1633e Jul 06 17:05:04.955059 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:04.955031 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 06 17:05:05.886163 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:05.886128 2563 generic.go:356] "Generic (PLEG): container finished" podID="ac602977-0938-43c7-8c38-8a511bb6f175" containerID="3562e6529b6cef22eb7ea7b0ddf116652b728a2fb73570ab92c99364cb11f72d" exitCode=0 Jul 06 17:05:05.886536 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:05.886170 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-140473332468848" event={"ID":"ac602977-0938-43c7-8c38-8a511bb6f175","Type":"ContainerDied","Data":"3562e6529b6cef22eb7ea7b0ddf116652b728a2fb73570ab92c99364cb11f72d"} Jul 06 17:05:05.886536 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:05.886202 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7036-140473332468848" event={"ID":"ac602977-0938-43c7-8c38-8a511bb6f175","Type":"ContainerStarted","Data":"039860fd15f707535af3d52ced7bce57dc12084b5179e8acc0f7b7b0feb1633e"} Jul 06 17:05:07.029923 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.029894 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:07.108163 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.108130 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7036-140473332468848_ac602977-0938-43c7-8c38-8a511bb6f175/test-curl-7036-140473332468848/0.log" Jul 06 17:05:07.137546 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.137508 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7036-140473332468848"] Jul 06 17:05:07.145614 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.145537 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7036-140473332468848"] Jul 06 17:05:07.148810 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.148771 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hm92f\" (UniqueName: \"kubernetes.io/projected/ac602977-0938-43c7-8c38-8a511bb6f175-kube-api-access-hm92f\") pod \"ac602977-0938-43c7-8c38-8a511bb6f175\" (UID: \"ac602977-0938-43c7-8c38-8a511bb6f175\") " Jul 06 17:05:07.150799 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.150760 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac602977-0938-43c7-8c38-8a511bb6f175-kube-api-access-hm92f" (OuterVolumeSpecName: "kube-api-access-hm92f") pod "ac602977-0938-43c7-8c38-8a511bb6f175" (UID: "ac602977-0938-43c7-8c38-8a511bb6f175"). InnerVolumeSpecName "kube-api-access-hm92f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:05:07.250106 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.250068 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm92f\" (UniqueName: \"kubernetes.io/projected/ac602977-0938-43c7-8c38-8a511bb6f175-kube-api-access-hm92f\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:05:07.794032 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.794002 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac602977-0938-43c7-8c38-8a511bb6f175" path="/var/lib/kubelet/pods/ac602977-0938-43c7-8c38-8a511bb6f175/volumes" Jul 06 17:05:07.897218 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.897188 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7036-140473332468848" Jul 06 17:05:07.897367 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:07.897202 2563 scope.go:117] "RemoveContainer" containerID="3562e6529b6cef22eb7ea7b0ddf116652b728a2fb73570ab92c99364cb11f72d" Jul 06 17:05:54.542653 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:54.542621 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj"] Jul 06 17:05:54.543167 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:54.542899 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" containerID="cri-o://4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5" gracePeriod=30 Jul 06 17:05:59.802959 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.802899 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 17:05:59.945748 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.945717 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-data\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.945748 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.945750 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-envoy\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946016 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.945776 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-podinfo\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946016 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.945823 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-token\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946016 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.945902 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-socket\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946016 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.945955 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-certs\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946016 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946005 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istiod-ca-cert\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946290 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946034 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-credential-socket\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946290 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946058 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6hwwt\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-kube-api-access-6hwwt\") pod \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\" (UID: \"6c7de4a6-9f91-44f3-84fc-7a5793fc157d\") " Jul 06 17:05:59.946290 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946233 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:05:59.946290 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946226 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:05:59.946499 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946460 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:05:59.946499 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946462 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:05:59.946499 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946386 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:05:59.946499 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946500 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:05:59.946706 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.946563 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-data" (OuterVolumeSpecName: "istio-data") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:05:59.948267 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.948239 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:05:59.948376 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.948261 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-token" (OuterVolumeSpecName: "istio-token") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:05:59.948376 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.948289 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-kube-api-access-6hwwt" (OuterVolumeSpecName: "kube-api-access-6hwwt") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "kube-api-access-6hwwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:05:59.948376 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:05:59.948355 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6c7de4a6-9f91-44f3-84fc-7a5793fc157d" (UID: "6c7de4a6-9f91-44f3-84fc-7a5793fc157d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:06:00.047785 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047744 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.047785 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047784 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.048032 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047824 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hwwt\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-kube-api-access-6hwwt\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.048032 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047838 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.048032 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047850 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.048032 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047862 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.048032 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.047872 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7de4a6-9f91-44f3-84fc-7a5793fc157d-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:00.159312 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.159276 2563 generic.go:356] "Generic (PLEG): container finished" podID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerID="4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5" exitCode=0 Jul 06 17:06:00.159485 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.159338 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" Jul 06 17:06:00.159485 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.159354 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" event={"ID":"6c7de4a6-9f91-44f3-84fc-7a5793fc157d","Type":"ContainerDied","Data":"4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5"} Jul 06 17:06:00.159485 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.159438 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj" event={"ID":"6c7de4a6-9f91-44f3-84fc-7a5793fc157d","Type":"ContainerDied","Data":"718792046d422bcde5d84bdc3854e076ebad11f8a2ed19d985feef8b5eb0b4db"} Jul 06 17:06:00.159485 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.159462 2563 scope.go:117] "RemoveContainer" containerID="4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5" Jul 06 17:06:00.168971 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.168955 2563 scope.go:117] "RemoveContainer" containerID="80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a" Jul 06 17:06:00.177078 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.177062 2563 scope.go:117] "RemoveContainer" containerID="4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5" Jul 06 17:06:00.177326 ip-10-0-136-202 kubenswrapper[2563]: E0706 17:06:00.177306 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5\": container with ID starting with 4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5 not found: ID does not exist" containerID="4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5" Jul 06 17:06:00.177377 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.177335 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5"} err="failed to get container status \"4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5\": rpc error: code = NotFound desc = could not find container \"4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5\": container with ID starting with 4ebde95dab338f23deb7da40d110b7f82847dcc2ae9fcf0bc3588a2d70c7bdc5 not found: ID does not exist" Jul 06 17:06:00.177377 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.177352 2563 scope.go:117] "RemoveContainer" containerID="80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a" Jul 06 17:06:00.177565 ip-10-0-136-202 kubenswrapper[2563]: E0706 17:06:00.177545 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a\": container with ID starting with 80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a not found: ID does not exist" containerID="80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a" Jul 06 17:06:00.177612 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.177571 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a"} err="failed to get container status \"80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a\": rpc error: code = NotFound desc = could not find container \"80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a\": container with ID starting with 80273ef3d0d7a8e9b01bd711a920cf15240570f9f2ec543c9a94034314433e7a not found: ID does not exist" Jul 06 17:06:00.192267 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.192211 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj"] Jul 06 17:06:00.194622 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:00.194603 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-0e87f3-openshift-default-6fbc9b7855-5nvcj"] Jul 06 17:06:01.793383 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:01.793349 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" path="/var/lib/kubelet/pods/6c7de4a6-9f91-44f3-84fc-7a5793fc157d/volumes" Jul 06 17:06:03.021562 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:03.021529 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj"] Jul 06 17:06:03.022083 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:03.021748 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" containerID="cri-o://51ca34744f7f10fb934bcd72f96f38e41f0dcb7c17fbfbc96ffe303a743e877a" gracePeriod=30 Jul 06 17:06:08.200729 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.200697 2563 generic.go:356] "Generic (PLEG): container finished" podID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerID="51ca34744f7f10fb934bcd72f96f38e41f0dcb7c17fbfbc96ffe303a743e877a" exitCode=0 Jul 06 17:06:08.201117 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.200768 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" event={"ID":"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da","Type":"ContainerDied","Data":"51ca34744f7f10fb934bcd72f96f38e41f0dcb7c17fbfbc96ffe303a743e877a"} Jul 06 17:06:08.201117 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.200847 2563 scope.go:117] "RemoveContainer" containerID="bb318dc895d92952936f7240d0f2f8f6d780c7f96991e6060a47b216db62afdb" Jul 06 17:06:08.279275 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.279250 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 17:06:08.328861 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328746 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-data\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.328861 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328784 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-certs\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.328861 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328838 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-credential-socket\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329188 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328872 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-token\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329188 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328913 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-podinfo\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329188 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328959 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f6swq\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-kube-api-access-f6swq\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329188 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.328990 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-envoy\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329188 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329018 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-socket\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329188 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329064 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istiod-ca-cert\") pod \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\" (UID: \"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da\") " Jul 06 17:06:08.329525 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329057 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:06:08.329525 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329143 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:06:08.329525 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329335 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:06:08.329689 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329574 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 06 17:06:08.329689 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.329597 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-data" (OuterVolumeSpecName: "istio-data") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:06:08.331403 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.331368 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 06 17:06:08.331535 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.331426 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-kube-api-access-f6swq" (OuterVolumeSpecName: "kube-api-access-f6swq") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "kube-api-access-f6swq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:06:08.331594 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.331579 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-token" (OuterVolumeSpecName: "istio-token") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 06 17:06:08.331653 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.331632 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" (UID: "4024fa4d-7e3a-4079-ac5e-d3009cdbe2da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 06 17:06:08.430517 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430481 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-data\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430517 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430511 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-certs\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430517 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430519 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-credential-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430517 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430528 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-token\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430787 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430536 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-podinfo\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430787 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430544 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6swq\" (UniqueName: \"kubernetes.io/projected/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-kube-api-access-f6swq\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430787 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430553 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istio-envoy\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430787 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430561 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-workload-socket\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.430787 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.430568 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da-istiod-ca-cert\") on node \"ip-10-0-136-202.ec2.internal\" DevicePath \"\"" Jul 06 17:06:08.484548 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.484517 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kf4b4_b3b81d55-e8b2-4502-b8dc-6844ea4289f7/manager/0.log" Jul 06 17:06:08.605519 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.605493 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-47ctp_ef614215-9aed-4c52-b811-3c5612cde758/manager/0.log" Jul 06 17:06:08.865128 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:08.865039 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722605-mxqnr_1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41/cleanup/0.log" Jul 06 17:06:09.107749 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.107717 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66fcc46d69-wprqg_320b113e-8b5d-4622-8c88-6f4c3fd28905/manager/0.log" Jul 06 17:06:09.207616 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.207530 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" Jul 06 17:06:09.207616 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.207542 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj" event={"ID":"4024fa4d-7e3a-4079-ac5e-d3009cdbe2da","Type":"ContainerDied","Data":"2a681ac7a2635189840f8c48f02dd4f726d8e593c754af13a34544bd97fdc94a"} Jul 06 17:06:09.207616 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.207591 2563 scope.go:117] "RemoveContainer" containerID="51ca34744f7f10fb934bcd72f96f38e41f0dcb7c17fbfbc96ffe303a743e877a" Jul 06 17:06:09.230631 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.230605 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p8tjw_39b64fcc-6538-4001-9c0e-f426eef87f30/server/0.log" Jul 06 17:06:09.240256 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.240231 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj"] Jul 06 17:06:09.244273 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.244251 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-327802-openshift-default-f989755c9-8qvfj"] Jul 06 17:06:09.359634 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.359603 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-62n4t_70cb66d3-9dc3-464d-8b61-4945c1270a9d/manager/0.log" Jul 06 17:06:09.794255 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.794214 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" path="/var/lib/kubelet/pods/4024fa4d-7e3a-4079-ac5e-d3009cdbe2da/volumes" Jul 06 17:06:09.796620 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.796598 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7896b945bc-mjhzv_d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2/copy-manifests/0.log" Jul 06 17:06:09.814359 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.814336 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7896b945bc-mjhzv_d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2/manager/0.log" Jul 06 17:06:09.934524 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:09.934493 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-wmhxp_076b7c9d-4a18-4de3-aa35-ebd1680a0729/postgres/0.log" Jul 06 17:06:11.483350 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:11.483322 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kqlh7_db33abe1-697d-49ff-889e-4ff114f74ce5/manager/0.log" Jul 06 17:06:11.604156 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:11.604122 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4sxp9_745d1b0e-1e28-4b5d-883e-debffe68b46f/manager/0.log" Jul 06 17:06:11.736487 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:11.736413 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9hp2k_12c6b817-4094-4629-b554-3ef3b9f714cc/kuadrant-console-plugin/0.log" Jul 06 17:06:11.869109 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:11.869080 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7lwdw_f054c648-4aa7-4c32-bc3a-e5cb5a04033f/registry-server/0.log" Jul 06 17:06:12.305804 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:12.305755 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tvrm8_91c55469-a3bd-4def-9aaa-7d505c6cf825/manager/0.log" Jul 06 17:06:12.692554 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:12.692521 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8grh_566106b0-3f8d-453b-8c41-7f21e1154b70/istio-proxy/0.log" Jul 06 17:06:13.120439 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:13.120406 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5846db9b4f-nkmxq_63fed56d-b091-4741-8cdc-5e47db6ed80d/kube-auth-proxy/0.log" Jul 06 17:06:13.260942 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:13.260913 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/1.log" Jul 06 17:06:13.555237 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:13.555151 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dsvxc_9c32da6e-f25c-4c71-a97e-db398d6116ef/payload-processing/0.log" Jul 06 17:06:13.681871 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:13.681839 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996dbbc49-g98f5_d1148015-927a-46cb-8fa4-e4520fc74763/router/0.log" Jul 06 17:06:14.051655 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:14.051625 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-spp5d_6982831b-656b-4046-91a8-26e5eede3b00/main/0.log" Jul 06 17:06:14.312415 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:14.312330 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-7mtgw_7441d830-e5de-4cf1-a6db-7b07d0381955/main/0.log" Jul 06 17:06:14.574539 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:14.574458 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-b2nqj_f5e0fa44-5e38-4a19-bb80-f8e70ea2c59b/main/0.log" Jul 06 17:06:18.875827 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.875783 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-842gk/must-gather-55685"] Jul 06 17:06:18.876247 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876234 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" Jul 06 17:06:18.876298 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876249 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" Jul 06 17:06:18.876298 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876257 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac602977-0938-43c7-8c38-8a511bb6f175" containerName="test-curl-7036-140473332468848" Jul 06 17:06:18.876298 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876263 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac602977-0938-43c7-8c38-8a511bb6f175" containerName="test-curl-7036-140473332468848" Jul 06 17:06:18.876390 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876301 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" Jul 06 17:06:18.876390 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876307 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" Jul 06 17:06:18.876390 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876315 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" Jul 06 17:06:18.876390 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876327 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" Jul 06 17:06:18.876512 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876402 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" Jul 06 17:06:18.876512 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876412 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" Jul 06 17:06:18.876512 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876417 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" Jul 06 17:06:18.876512 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876425 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7de4a6-9f91-44f3-84fc-7a5793fc157d" containerName="istio-proxy" Jul 06 17:06:18.876512 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876430 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac602977-0938-43c7-8c38-8a511bb6f175" containerName="test-curl-7036-140473332468848" Jul 06 17:06:18.876660 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876514 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" Jul 06 17:06:18.876660 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.876526 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4024fa4d-7e3a-4079-ac5e-d3009cdbe2da" containerName="istio-proxy" Jul 06 17:06:18.884447 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.884411 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:18.892780 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.892759 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-842gk\"/\"kube-root-ca.crt\"" Jul 06 17:06:18.898168 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.898153 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-842gk\"/\"openshift-service-ca.crt\"" Jul 06 17:06:18.901292 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.901273 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-842gk\"/\"default-dockercfg-zbpwj\"" Jul 06 17:06:18.912467 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.912448 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-842gk/must-gather-55685"] Jul 06 17:06:18.934473 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.934446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8fv\" (UniqueName: \"kubernetes.io/projected/daf5a793-746a-4731-b9b1-5eaebdfe9826-kube-api-access-dk8fv\") pod \"must-gather-55685\" (UID: \"daf5a793-746a-4731-b9b1-5eaebdfe9826\") " pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:18.934594 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:18.934485 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/daf5a793-746a-4731-b9b1-5eaebdfe9826-must-gather-output\") pod \"must-gather-55685\" (UID: \"daf5a793-746a-4731-b9b1-5eaebdfe9826\") " pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:19.035334 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:19.035295 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8fv\" (UniqueName: \"kubernetes.io/projected/daf5a793-746a-4731-b9b1-5eaebdfe9826-kube-api-access-dk8fv\") pod \"must-gather-55685\" (UID: \"daf5a793-746a-4731-b9b1-5eaebdfe9826\") " pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:19.035334 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:19.035340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/daf5a793-746a-4731-b9b1-5eaebdfe9826-must-gather-output\") pod \"must-gather-55685\" (UID: \"daf5a793-746a-4731-b9b1-5eaebdfe9826\") " pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:19.035664 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:19.035649 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/daf5a793-746a-4731-b9b1-5eaebdfe9826-must-gather-output\") pod \"must-gather-55685\" (UID: \"daf5a793-746a-4731-b9b1-5eaebdfe9826\") " pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:19.048225 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:19.048200 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8fv\" (UniqueName: \"kubernetes.io/projected/daf5a793-746a-4731-b9b1-5eaebdfe9826-kube-api-access-dk8fv\") pod \"must-gather-55685\" (UID: \"daf5a793-746a-4731-b9b1-5eaebdfe9826\") " pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:19.193776 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:19.193691 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-842gk/must-gather-55685" Jul 06 17:06:19.406916 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:19.406852 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-842gk/must-gather-55685"] Jul 06 17:06:19.409142 ip-10-0-136-202 kubenswrapper[2563]: W0706 17:06:19.409115 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddaf5a793_746a_4731_b9b1_5eaebdfe9826.slice/crio-a5a293dd95cb330a8d513d94214822a98854e12315d89dbfe0e32444fe2375b3 WatchSource:0}: Error finding container a5a293dd95cb330a8d513d94214822a98854e12315d89dbfe0e32444fe2375b3: Status 404 returned error can't find the container with id a5a293dd95cb330a8d513d94214822a98854e12315d89dbfe0e32444fe2375b3 Jul 06 17:06:20.268946 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:20.268911 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-842gk/must-gather-55685" event={"ID":"daf5a793-746a-4731-b9b1-5eaebdfe9826","Type":"ContainerStarted","Data":"a5a293dd95cb330a8d513d94214822a98854e12315d89dbfe0e32444fe2375b3"} Jul 06 17:06:21.279884 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:21.279846 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-842gk/must-gather-55685" event={"ID":"daf5a793-746a-4731-b9b1-5eaebdfe9826","Type":"ContainerStarted","Data":"49140d765849ae88a4bfd1dbb16228f9abb11dd2ece1664c28ee97f49c49668e"} Jul 06 17:06:21.280389 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:21.279899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-842gk/must-gather-55685" event={"ID":"daf5a793-746a-4731-b9b1-5eaebdfe9826","Type":"ContainerStarted","Data":"b3413b69b1b172ea3cb289501d181d762c73ee9535810ff9893ca4eb2e4946a9"} Jul 06 17:06:22.343010 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:22.342982 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-m285v_65242094-074d-41d5-a8e9-d22a5c636b9f/global-pull-secret-syncer/0.log" Jul 06 17:06:22.621515 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:22.621416 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zhcw4_0a8a26bd-3e9b-4308-915d-f6fc9cd8394a/konnectivity-agent/0.log" Jul 06 17:06:22.694569 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:22.694539 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-202.ec2.internal_a35c3367aea0364e635c7dba529cb3ff/haproxy/0.log" Jul 06 17:06:27.579829 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:27.579771 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-kqlh7_db33abe1-697d-49ff-889e-4ff114f74ce5/manager/0.log" Jul 06 17:06:27.635230 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:27.635198 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4sxp9_745d1b0e-1e28-4b5d-883e-debffe68b46f/manager/0.log" Jul 06 17:06:27.670544 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:27.670511 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-9hp2k_12c6b817-4094-4629-b554-3ef3b9f714cc/kuadrant-console-plugin/0.log" Jul 06 17:06:27.721460 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:27.721422 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7lwdw_f054c648-4aa7-4c32-bc3a-e5cb5a04033f/registry-server/0.log" Jul 06 17:06:28.047813 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:28.045992 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-tvrm8_91c55469-a3bd-4def-9aaa-7d505c6cf825/manager/0.log" Jul 06 17:06:30.139842 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:30.139808 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-jk7mk_ba335293-eebb-4b84-87aa-84258e6c4111/cluster-monitoring-operator/0.log" Jul 06 17:06:30.502070 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:30.501992 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ld8fk_e4a0d007-fab8-41f5-abe1-25ea08c88780/node-exporter/0.log" Jul 06 17:06:30.537595 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:30.537513 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ld8fk_e4a0d007-fab8-41f5-abe1-25ea08c88780/kube-rbac-proxy/0.log" Jul 06 17:06:30.569807 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:30.569753 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ld8fk_e4a0d007-fab8-41f5-abe1-25ea08c88780/init-textfile/0.log" Jul 06 17:06:31.185388 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:31.185363 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-kjvrs_31373c34-0755-4552-8ed3-0570d9fd37db/prometheus-operator-admission-webhook/0.log" Jul 06 17:06:31.978296 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:31.978229 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-842gk/must-gather-55685" podStartSLOduration=13.066521701 podStartE2EDuration="13.978205539s" totalImagesPullingTime="911.683838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-06 17:06:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:06:21.313951346 +0000 UTC m=+2564.029170314" watchObservedRunningTime="2026-07-06 17:06:31.978205539 +0000 UTC m=+2574.693424513" Jul 06 17:06:31.980723 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:31.980696 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c"] Jul 06 17:06:31.989218 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:31.989193 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:31.997768 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:31.997738 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c"] Jul 06 17:06:32.078654 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.078621 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-proc\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.078878 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.078663 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-lib-modules\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.078878 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.078695 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-sys\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.078878 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.078726 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-podres\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.078878 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.078757 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d782q\" (UniqueName: \"kubernetes.io/projected/2562823c-4418-488d-87fe-663273760430-kube-api-access-d782q\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180292 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-proc\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180292 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180301 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-lib-modules\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180331 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-sys\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180362 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-podres\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180385 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-proc\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180390 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d782q\" (UniqueName: \"kubernetes.io/projected/2562823c-4418-488d-87fe-663273760430-kube-api-access-d782q\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180458 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-lib-modules\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180458 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-sys\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.180550 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.180540 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/2562823c-4418-488d-87fe-663273760430-podres\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.191380 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.191344 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d782q\" (UniqueName: \"kubernetes.io/projected/2562823c-4418-488d-87fe-663273760430-kube-api-access-d782q\") pod \"perf-node-gather-daemonset-lrt4c\" (UID: \"2562823c-4418-488d-87fe-663273760430\") " pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.302646 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.302573 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:32.490578 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.489205 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c"] Jul 06 17:06:32.887784 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:32.887754 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-xpvrt_e51b6357-87fc-4d48-a162-054f00096e6b/networking-console-plugin/0.log" Jul 06 17:06:33.372436 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:33.372388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" event={"ID":"2562823c-4418-488d-87fe-663273760430","Type":"ContainerStarted","Data":"8b0a7965ede0d515101d1f45a856cf0defff20d3d252e0ba860538a00f27aee6"} Jul 06 17:06:33.372965 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:33.372472 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" event={"ID":"2562823c-4418-488d-87fe-663273760430","Type":"ContainerStarted","Data":"5eae8f733a244541e1657c3f3437289728e32494718d12ce7c69ca0bf3d902bb"} Jul 06 17:06:33.372965 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:33.372889 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:35.764031 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:35.764002 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qtdqx_a877b735-824f-46f9-b2e4-016dc8bd2725/dns/0.log" Jul 06 17:06:35.789514 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:35.789489 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qtdqx_a877b735-824f-46f9-b2e4-016dc8bd2725/kube-rbac-proxy/0.log" Jul 06 17:06:35.900040 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:35.900011 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nm6dl_b1d64b6b-9b5a-4eb1-88db-e29385e6faa7/dns-node-resolver/0.log" Jul 06 17:06:36.635985 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:36.635956 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-796kr_bede19a7-4918-4b48-8bf0-f5ade4d53795/node-ca/0.log" Jul 06 17:06:37.628582 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:37.628543 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8grh_566106b0-3f8d-453b-8c41-7f21e1154b70/istio-proxy/0.log" Jul 06 17:06:37.886874 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:37.886775 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5846db9b4f-nkmxq_63fed56d-b091-4741-8cdc-5e47db6ed80d/kube-auth-proxy/0.log" Jul 06 17:06:37.921482 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:37.921455 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/0.log" Jul 06 17:06:37.956676 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:37.956642 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/1.log" Jul 06 17:06:38.074377 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:38.074344 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dsvxc_9c32da6e-f25c-4c71-a97e-db398d6116ef/payload-processing/0.log" Jul 06 17:06:38.103107 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:38.103077 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996dbbc49-g98f5_d1148015-927a-46cb-8fa4-e4520fc74763/router/0.log" Jul 06 17:06:38.777162 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:38.777138 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5p5zj_73c3b173-35b2-41ac-bd6f-e568f1229742/serve-healthcheck-canary/0.log" Jul 06 17:06:39.340180 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.340143 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-nrkt5_d32c3f81-2740-490c-b120-f67350df8f95/insights-operator/0.log" Jul 06 17:06:39.340759 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.340737 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-nrkt5_d32c3f81-2740-490c-b120-f67350df8f95/insights-operator/1.log" Jul 06 17:06:39.388353 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.388324 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" Jul 06 17:06:39.409533 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.409474 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-842gk/perf-node-gather-daemonset-lrt4c" podStartSLOduration=8.409453818 podStartE2EDuration="8.409453818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-06 17:06:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-06 17:06:33.411309402 +0000 UTC m=+2576.126528376" watchObservedRunningTime="2026-07-06 17:06:39.409453818 +0000 UTC m=+2582.124672795" Jul 06 17:06:39.450124 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.450097 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdkwl_3238e000-945c-4464-aa32-2f2e0843d959/kube-rbac-proxy/0.log" Jul 06 17:06:39.477129 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.477101 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdkwl_3238e000-945c-4464-aa32-2f2e0843d959/exporter/0.log" Jul 06 17:06:39.506581 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:39.506557 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sdkwl_3238e000-945c-4464-aa32-2f2e0843d959/extractor/0.log" Jul 06 17:06:42.291788 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.291749 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kf4b4_b3b81d55-e8b2-4502-b8dc-6844ea4289f7/manager/0.log" Jul 06 17:06:42.384051 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.383983 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8grh_566106b0-3f8d-453b-8c41-7f21e1154b70/istio-proxy/0.log" Jul 06 17:06:42.466695 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.466665 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-47ctp_ef614215-9aed-4c52-b811-3c5612cde758/manager/0.log" Jul 06 17:06:42.640090 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.640065 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29722605-mxqnr_1f2d0c15-3f29-4ed1-a31b-6c72c65f9e41/cleanup/0.log" Jul 06 17:06:42.656657 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.656621 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5846db9b4f-nkmxq_63fed56d-b091-4741-8cdc-5e47db6ed80d/kube-auth-proxy/0.log" Jul 06 17:06:42.678474 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.678447 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/0.log" Jul 06 17:06:42.721155 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.721127 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/1.log" Jul 06 17:06:42.826578 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.826547 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-66fcc46d69-wprqg_320b113e-8b5d-4622-8c88-6f4c3fd28905/manager/0.log" Jul 06 17:06:42.843009 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.842979 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dsvxc_9c32da6e-f25c-4c71-a97e-db398d6116ef/payload-processing/0.log" Jul 06 17:06:42.863970 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.863941 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996dbbc49-g98f5_d1148015-927a-46cb-8fa4-e4520fc74763/router/0.log" Jul 06 17:06:42.869453 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.869430 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p8tjw_39b64fcc-6538-4001-9c0e-f426eef87f30/server/0.log" Jul 06 17:06:42.980967 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:42.980506 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-62n4t_70cb66d3-9dc3-464d-8b61-4945c1270a9d/manager/0.log" Jul 06 17:06:43.292411 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:43.292384 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7896b945bc-mjhzv_d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2/manager/0.log" Jul 06 17:06:43.320764 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:43.320742 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7896b945bc-mjhzv_d5b5b93d-5d4b-42a1-b3bc-2f5ae4bde4b2/copy-manifests/0.log" Jul 06 17:06:43.356148 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:43.356101 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-wmhxp_076b7c9d-4a18-4de3-aa35-ebd1680a0729/postgres/0.log" Jul 06 17:06:44.179742 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:44.179713 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-7f964fb678f8grh_566106b0-3f8d-453b-8c41-7f21e1154b70/istio-proxy/0.log" Jul 06 17:06:44.592081 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:44.591511 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5846db9b4f-nkmxq_63fed56d-b091-4741-8cdc-5e47db6ed80d/kube-auth-proxy/0.log" Jul 06 17:06:44.666806 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:44.666760 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/0.log" Jul 06 17:06:44.701149 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:44.701110 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5896d4dd49-4ndln_d446c722-3560-485f-9503-7be061763145/istio-proxy/1.log" Jul 06 17:06:44.886102 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:44.886024 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-dsvxc_9c32da6e-f25c-4c71-a97e-db398d6116ef/payload-processing/0.log" Jul 06 17:06:44.909509 ip-10-0-136-202 kubenswrapper[2563]: I0706 17:06:44.909482 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-996dbbc49-g98f5_d1148015-927a-46cb-8fa4-e4520fc74763/router/0.log"