Jul 24 23:29:00.182506 ip-10-0-131-63 systemd[1]: Starting Kubernetes Kubelet... Jul 24 23:29:00.615590 ip-10-0-131-63 kubenswrapper[2586]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 23:29:00.615590 ip-10-0-131-63 kubenswrapper[2586]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 23:29:00.615590 ip-10-0-131-63 kubenswrapper[2586]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 23:29:00.615590 ip-10-0-131-63 kubenswrapper[2586]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 23:29:00.615590 ip-10-0-131-63 kubenswrapper[2586]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 23:29:00.616676 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.616603 2586 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 23:29:00.621462 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621446 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 23:29:00.621462 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621463 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621467 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621471 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621474 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621477 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621480 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621483 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621486 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621489 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621492 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621495 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621498 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621501 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621505 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621508 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621510 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621513 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621516 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621519 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621522 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 23:29:00.621524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621525 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621528 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621531 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621534 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621538 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621543 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621545 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621548 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621550 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621553 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621556 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621558 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621561 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621563 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621566 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621568 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621571 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621573 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621576 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621578 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 23:29:00.621995 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621581 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621583 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621586 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621589 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621592 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621596 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621599 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621602 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621605 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621608 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621611 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621613 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621616 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621619 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621623 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621626 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621628 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621631 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621633 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 23:29:00.622496 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621636 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621639 2586 feature_gate.go:328] unrecognized feature gate: Example Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621642 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621644 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621647 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621649 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621652 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621654 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621657 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621659 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621662 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621665 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621668 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621671 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621673 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621676 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621679 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621681 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621684 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621686 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 23:29:00.622949 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621689 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621691 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621694 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621696 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621699 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621701 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621703 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621706 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621709 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621712 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621714 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621717 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621719 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621722 2586 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621725 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621727 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621730 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621732 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.621735 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622432 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 23:29:00.623443 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622440 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622443 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622446 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622449 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622452 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622455 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622457 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622460 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622463 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622466 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622469 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622472 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622474 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622477 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622480 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622482 2586 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622485 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622487 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622490 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622493 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 23:29:00.623970 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622496 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622499 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622501 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622504 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622507 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622510 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622513 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622515 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622518 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622521 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622523 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622526 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622529 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622531 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622534 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622537 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622540 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622542 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622545 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622547 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 23:29:00.624572 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622550 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622552 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622555 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622558 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622560 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622563 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622565 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622568 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622570 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622573 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622576 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622578 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622581 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622583 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622586 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622588 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622591 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622593 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622596 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622598 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 23:29:00.625056 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622601 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622604 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622606 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622609 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622611 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622613 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622616 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622621 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622624 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622627 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622630 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622633 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622635 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622638 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622640 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622648 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622650 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622653 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622656 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622658 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 23:29:00.625559 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622661 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622663 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622666 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622669 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622671 2586 feature_gate.go:328] unrecognized feature gate: Example Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622674 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622677 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622681 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622685 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622687 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622690 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622692 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622695 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622697 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622700 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622702 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622706 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.622710 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623076 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 23:29:00.626054 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623081 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623085 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623088 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623092 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623095 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623097 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623100 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623103 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623109 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623112 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623114 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623117 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623119 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623122 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623125 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623127 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623130 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623132 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623135 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623137 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 23:29:00.626524 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623140 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623142 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623145 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623147 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623150 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623153 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623156 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623158 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623161 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623165 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623168 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623171 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623174 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623177 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623180 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623183 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623185 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623188 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623191 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623194 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 23:29:00.626994 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623199 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623202 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623205 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623208 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623212 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623215 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623218 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623221 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623223 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623226 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623228 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623231 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623247 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623250 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623253 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623256 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623259 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623261 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623264 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 23:29:00.627507 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623266 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623269 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623271 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623274 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623276 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623279 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623281 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623284 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623286 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623289 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623291 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623294 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623296 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623300 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623302 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623305 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623308 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623311 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623314 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623316 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 23:29:00.627967 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623319 2586 feature_gate.go:328] unrecognized feature gate: Example Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623322 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623325 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623328 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623330 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623334 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623337 2586 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623342 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623345 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623347 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623350 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623353 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623355 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623358 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623360 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623363 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623365 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623368 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.623371 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623431 2586 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623438 2586 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 23:29:00.628491 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623445 2586 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623449 2586 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623454 2586 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623457 2586 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623464 2586 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623468 2586 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623471 2586 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623475 2586 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623479 2586 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623482 2586 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623485 2586 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623488 2586 flags.go:64] FLAG: --cgroup-root="" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623491 2586 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623494 2586 flags.go:64] FLAG: --client-ca-file="" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623497 2586 flags.go:64] FLAG: --cloud-provider="external" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623500 2586 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623505 2586 flags.go:64] FLAG: --cluster-domain="" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623508 2586 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623512 2586 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623515 2586 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623519 2586 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623522 2586 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623525 2586 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 23:29:00.629033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623528 2586 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623532 2586 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623535 2586 flags.go:64] FLAG: --contention-profiling="false" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623538 2586 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623541 2586 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623544 2586 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623548 2586 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623552 2586 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623555 2586 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623558 2586 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623561 2586 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623564 2586 flags.go:64] FLAG: --enable-server="true" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623567 2586 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623573 2586 flags.go:64] FLAG: --event-burst="100" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623576 2586 flags.go:64] FLAG: --event-qps="50" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623579 2586 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623582 2586 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623585 2586 flags.go:64] FLAG: --eviction-hard="" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623589 2586 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623592 2586 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623595 2586 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623598 2586 flags.go:64] FLAG: --eviction-soft="" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623601 2586 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623604 2586 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623607 2586 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 23:29:00.629599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623610 2586 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623613 2586 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623617 2586 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623620 2586 flags.go:64] FLAG: --feature-gates="" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623624 2586 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623627 2586 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623630 2586 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623633 2586 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623636 2586 flags.go:64] FLAG: --healthz-port="10248" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623639 2586 flags.go:64] FLAG: --help="false" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623642 2586 flags.go:64] FLAG: --hostname-override="ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623645 2586 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623648 2586 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623650 2586 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623654 2586 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623657 2586 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623660 2586 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623663 2586 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623666 2586 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623668 2586 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623672 2586 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623676 2586 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623679 2586 flags.go:64] FLAG: --kube-reserved="" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623681 2586 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 23:29:00.630212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623684 2586 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623688 2586 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623691 2586 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623693 2586 flags.go:64] FLAG: --lock-file="" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623696 2586 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623699 2586 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623702 2586 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623707 2586 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623710 2586 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623713 2586 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623717 2586 flags.go:64] FLAG: --logging-format="text" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623719 2586 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623723 2586 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623726 2586 flags.go:64] FLAG: --manifest-url="" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623728 2586 flags.go:64] FLAG: --manifest-url-header="" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623733 2586 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623736 2586 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623740 2586 flags.go:64] FLAG: --max-pods="110" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623742 2586 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623746 2586 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623748 2586 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623751 2586 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623754 2586 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623757 2586 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623760 2586 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 23:29:00.631119 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623767 2586 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623770 2586 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623773 2586 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623777 2586 flags.go:64] FLAG: --pod-cidr="" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623779 2586 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623785 2586 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623788 2586 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623791 2586 flags.go:64] FLAG: --pods-per-core="0" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623794 2586 flags.go:64] FLAG: --port="10250" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623797 2586 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623800 2586 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0efd3ee16b0199300" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623804 2586 flags.go:64] FLAG: --qos-reserved="" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623807 2586 flags.go:64] FLAG: --read-only-port="10255" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623810 2586 flags.go:64] FLAG: --register-node="true" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623812 2586 flags.go:64] FLAG: --register-with-taints="" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623816 2586 flags.go:64] FLAG: --registry-burst="10" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623819 2586 flags.go:64] FLAG: --registry-qps="5" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623823 2586 flags.go:64] FLAG: --reserved-cpus="" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623826 2586 flags.go:64] FLAG: --reserved-memory="" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623829 2586 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623832 2586 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623835 2586 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623838 2586 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623841 2586 flags.go:64] FLAG: --runonce="false" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623844 2586 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 23:29:00.631915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623847 2586 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623850 2586 flags.go:64] FLAG: --seccomp-default="false" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623852 2586 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623864 2586 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623867 2586 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623871 2586 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623873 2586 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623877 2586 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623880 2586 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623883 2586 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623885 2586 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623890 2586 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623893 2586 flags.go:64] FLAG: --system-cgroups="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623896 2586 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623901 2586 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623904 2586 flags.go:64] FLAG: --tls-cert-file="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623907 2586 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623912 2586 flags.go:64] FLAG: --tls-min-version="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623915 2586 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623918 2586 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623921 2586 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623924 2586 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623926 2586 flags.go:64] FLAG: --v="2" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623930 2586 flags.go:64] FLAG: --version="false" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623935 2586 flags.go:64] FLAG: --vmodule="" Jul 24 23:29:00.632536 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623939 2586 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.623942 2586 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624049 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624053 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624056 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624059 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624062 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624065 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624067 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624070 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624072 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624075 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624078 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624080 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624082 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624085 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624088 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624091 2586 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624095 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624098 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 23:29:00.633178 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624100 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624103 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624105 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624110 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624113 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624117 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624120 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624123 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624126 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624129 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624131 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624134 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624137 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624140 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624142 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624145 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624147 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624150 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624153 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 23:29:00.633694 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624155 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624158 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624160 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624163 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624165 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624168 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624171 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624174 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624176 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624179 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624181 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624185 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624188 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624190 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624193 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624196 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624198 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624201 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624203 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 23:29:00.634149 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624206 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624209 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624211 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624214 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624216 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624219 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624222 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624224 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624227 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624229 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624232 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624248 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624252 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624255 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624259 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624262 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624265 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624267 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624270 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 23:29:00.634624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624272 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624275 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624279 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624282 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624285 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624288 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624291 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624294 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624296 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624299 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624301 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624304 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624307 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624309 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624312 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624315 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624317 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624320 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624323 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624325 2586 feature_gate.go:328] unrecognized feature gate: Example Jul 24 23:29:00.635083 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624328 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624330 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624334 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.624338 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.624343 2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.633137 2586 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.633152 2586 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.633996 2586 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.634003 2586 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634058 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634063 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634066 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634070 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634073 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 23:29:00.635581 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634076 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634079 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634083 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634086 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634089 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634092 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634095 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634097 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634100 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634102 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634105 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634108 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634110 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634113 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634115 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634118 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634121 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634123 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634126 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 23:29:00.635931 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634136 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634139 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634142 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634144 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634147 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634151 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634154 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634157 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634159 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634162 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634165 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634168 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634171 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634173 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634176 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634180 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634184 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634187 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634190 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634192 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 23:29:00.636400 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634195 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634197 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634200 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634203 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634205 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634208 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634211 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634213 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634216 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634218 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634221 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634225 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634228 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634231 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634246 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634249 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634253 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634256 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634259 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 23:29:00.636881 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634261 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634264 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634267 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634269 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634272 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634274 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634277 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634280 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634283 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634285 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634288 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634290 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634293 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634295 2586 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634298 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634301 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634304 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634307 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634309 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634312 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 23:29:00.637421 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634314 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634317 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634319 2586 feature_gate.go:328] unrecognized feature gate: Example Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634322 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634324 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634327 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634329 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634332 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634335 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634337 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634340 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634342 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634345 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634348 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634350 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 23:29:00.637897 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634353 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.634358 2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634464 2586 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634469 2586 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634473 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634476 2586 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634479 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634482 2586 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634485 2586 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634487 2586 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634490 2586 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634493 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634495 2586 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634498 2586 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634501 2586 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 23:29:00.638307 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634504 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634506 2586 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634509 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634511 2586 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634515 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634518 2586 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634520 2586 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634523 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634526 2586 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634528 2586 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634531 2586 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634534 2586 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634536 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634538 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634541 2586 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634543 2586 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634546 2586 feature_gate.go:328] unrecognized feature gate: Example Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634549 2586 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634551 2586 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634554 2586 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 23:29:00.638680 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634557 2586 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634559 2586 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634562 2586 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634564 2586 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634568 2586 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634571 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634573 2586 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634576 2586 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634578 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634581 2586 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634583 2586 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634586 2586 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634589 2586 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634591 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634594 2586 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634596 2586 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634599 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634602 2586 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634604 2586 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 23:29:00.639161 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634607 2586 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634609 2586 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634612 2586 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634614 2586 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634617 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634620 2586 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634622 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634625 2586 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634628 2586 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634630 2586 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634633 2586 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634636 2586 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634640 2586 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634643 2586 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634646 2586 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634650 2586 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634653 2586 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634656 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634659 2586 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 23:29:00.639624 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634662 2586 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634664 2586 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634666 2586 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634669 2586 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634671 2586 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634674 2586 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634676 2586 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634679 2586 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634682 2586 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634684 2586 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634687 2586 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634690 2586 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634692 2586 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634695 2586 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634697 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634700 2586 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634702 2586 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634705 2586 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634707 2586 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 23:29:00.640084 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634710 2586 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634712 2586 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634715 2586 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634717 2586 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634719 2586 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634722 2586 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634724 2586 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634727 2586 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:00.634729 2586 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.634734 2586 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.634831 2586 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.637133 2586 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.639565 2586 server.go:1019] "Starting client certificate rotation" Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.639660 2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 23:29:00.640557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.639689 2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 23:29:00.642644 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.642630 2586 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 23:29:00.643715 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.643698 2586 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 23:29:00.658415 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.658394 2586 log.go:25] "Validated CRI v1 runtime API" Jul 24 23:29:00.661129 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.661114 2586 log.go:25] "Validated CRI v1 image API" Jul 24 23:29:00.662018 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.661999 2586 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 23:29:00.666275 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.666250 2586 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7e8d651c-c44f-4205-a337-032d411e7090:/dev/nvme0n1p3 8853fa74-9bb2-4718-b4de-6e47a99346d9:/dev/nvme0n1p4] Jul 24 23:29:00.666348 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.666274 2586 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 23:29:00.671928 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.671908 2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 23:29:00.673982 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.673877 2586 manager.go:216] Machine: {Timestamp:2026-07-24 23:29:00.67267173 +0000 UTC m=+0.365512556 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097985 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21d6c9bbfb6a50b7cd286fb57b0cb4 SystemUUID:ec21d6c9-bbfb-6a50-b7cd-286fb57b0cb4 BootID:bb18b22b-61e6-4e8b-8be3-0430e1d3484a Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:a1:d4:dd:46:0f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:a1:d4:dd:46:0f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:6a:0e:f7:a4:57 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 23:29:00.673982 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.673979 2586 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 23:29:00.674094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.674050 2586 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 23:29:00.677763 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.677740 2586 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 23:29:00.677886 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.677765 2586 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 23:29:00.678896 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.678886 2586 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 23:29:00.678939 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.678898 2586 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 23:29:00.678939 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.678906 2586 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 23:29:00.679838 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.679828 2586 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 23:29:00.681343 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.681332 2586 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 23:29:00.683519 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683508 2586 state_mem.go:36] "Initialized new in-memory state store" Jul 24 23:29:00.683637 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683627 2586 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 23:29:00.683667 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683655 2586 kubelet.go:486] "Attempting to sync node with API server" Jul 24 23:29:00.683735 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683667 2586 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 23:29:00.683735 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683687 2586 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 23:29:00.683735 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683698 2586 kubelet.go:398] "Adding apiserver pod source" Jul 24 23:29:00.683735 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.683708 2586 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 23:29:00.684037 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.684026 2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 23:29:00.684075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.684044 2586 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 23:29:00.686622 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.686608 2586 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 23:29:00.689098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.689085 2586 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 23:29:00.689146 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.689116 2586 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 23:29:00.691222 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691208 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691225 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691246 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691255 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691266 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691271 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691279 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691285 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691292 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 23:29:00.691300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691297 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 23:29:00.691539 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691314 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 23:29:00.691539 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.691320 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 23:29:00.692177 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.692168 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 23:29:00.692177 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.692176 2586 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 23:29:00.695161 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.695140 2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 23:29:00.695256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.695181 2586 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 23:29:00.695337 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.695319 2586 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 23:29:00.695584 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.695571 2586 server.go:1268] "Started kubelet" Jul 24 23:29:00.695667 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.695640 2586 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 23:29:00.695667 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.695630 2586 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 23:29:00.695784 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.695684 2586 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 23:29:00.696231 ip-10-0-131-63 systemd[1]: Started Kubernetes Kubelet. Jul 24 23:29:00.697651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.697632 2586 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 23:29:00.700305 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.700293 2586 server.go:310] "Adding debug handlers to kubelet server" Jul 24 23:29:00.704827 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.703603 2586 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-63.ec2.internal.18c55d99630cba4a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-63.ec2.internal,UID:ip-10-0-131-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-63.ec2.internal,},FirstTimestamp:2026-07-24 23:29:00.695427658 +0000 UTC m=+0.388268477,LastTimestamp:2026-07-24 23:29:00.695427658 +0000 UTC m=+0.388268477,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-63.ec2.internal,}" Jul 24 23:29:00.705680 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.705664 2586 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 23:29:00.708196 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.708181 2586 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 23:29:00.708268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.708194 2586 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 23:29:00.708268 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.708255 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:00.708918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.708899 2586 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 23:29:00.708918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.708899 2586 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 23:29:00.708918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.708930 2586 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 23:29:00.709058 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709043 2586 reconstruct.go:97] "Volume reconstruction finished" Jul 24 23:29:00.709058 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709050 2586 reconciler.go:29] "Reconciler: start to sync state" Jul 24 23:29:00.709443 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709425 2586 factory.go:55] Registering systemd factory Jul 24 23:29:00.709443 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709446 2586 factory.go:223] Registration of the systemd container factory successfully Jul 24 23:29:00.709931 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709913 2586 factory.go:153] Registering CRI-O factory Jul 24 23:29:00.709931 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709932 2586 factory.go:223] Registration of the crio container factory successfully Jul 24 23:29:00.710079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.709980 2586 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 23:29:00.710079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.710001 2586 factory.go:103] Registering Raw factory Jul 24 23:29:00.710079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.710015 2586 manager.go:1196] Started watching for new ooms in manager Jul 24 23:29:00.710472 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.710443 2586 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 23:29:00.710546 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.710505 2586 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 23:29:00.710546 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.710535 2586 manager.go:349] Starting recovery of all containers Jul 24 23:29:00.714557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.714505 2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66mvn" Jul 24 23:29:00.719923 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.719850 2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-66mvn" Jul 24 23:29:00.725378 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.725353 2586 manager.go:354] Recovery completed Jul 24 23:29:00.729501 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.729488 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.731387 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731366 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.731483 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731400 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.731483 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731416 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.731983 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731968 2586 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 23:29:00.731983 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731980 2586 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 23:29:00.732082 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731989 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.732082 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.731994 2586 state_mem.go:36] "Initialized new in-memory state store" Jul 24 23:29:00.732608 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.732595 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.732664 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.732619 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.732664 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.732635 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.733889 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.733873 2586 policy_none.go:49] "None policy: Start" Jul 24 23:29:00.733965 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.733894 2586 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 23:29:00.733965 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.733908 2586 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 23:29:00.735206 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.735193 2586 policy_none.go:47] "Start" Jul 24 23:29:00.764975 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.764958 2586 manager.go:337] "Starting Device Plugin manager" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.764992 2586 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.765004 2586 server.go:87] "Starting device plugin registration server" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.765206 2586 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.765217 2586 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.765345 2586 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.765426 2586 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.765432 2586 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.765871 2586 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.765918 2586 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:00.779210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.778290 2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 23:29:00.779760 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.779743 2586 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 23:29:00.779802 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.779786 2586 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 23:29:00.779838 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.779808 2586 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 23:29:00.779890 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.779844 2586 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 23:29:00.781675 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.781654 2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 23:29:00.865785 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.865653 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.866464 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.866445 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.866555 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.866475 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.866555 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.866490 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.866555 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.866514 2586 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.875322 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.875307 2586 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.875401 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.875328 2586 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-63.ec2.internal\": node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:00.880209 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.880188 2586 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal"] Jul 24 23:29:00.880299 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.880264 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.880929 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.880914 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.880979 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.880939 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.880979 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.880953 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.882146 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882135 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.882274 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882261 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.882312 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882290 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.882733 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882718 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.882796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882747 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.882796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882722 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.882796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882758 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.882796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882773 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.882796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.882784 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.883805 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.883791 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.883865 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.883813 2586 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 23:29:00.884405 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.884392 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientMemory" Jul 24 23:29:00.884473 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.884410 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 23:29:00.884473 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.884419 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeHasSufficientPID" Jul 24 23:29:00.894202 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.894188 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:00.910640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.910619 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5e1e16665eaea43b449bdccc3b3d492-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"e5e1e16665eaea43b449bdccc3b3d492\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.910723 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.910653 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e1e16665eaea43b449bdccc3b3d492-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"e5e1e16665eaea43b449bdccc3b3d492\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.910723 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:00.910669 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a87cd2889e81c5df0b8771970533f7d1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-63.ec2.internal\" (UID: \"a87cd2889e81c5df0b8771970533f7d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.917770 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.917750 2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-63.ec2.internal\" not found" node="ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.921809 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.921793 2586 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-63.ec2.internal\" not found" node="ip-10-0-131-63.ec2.internal" Jul 24 23:29:00.994827 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:00.994806 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.011165 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.011149 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5e1e16665eaea43b449bdccc3b3d492-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"e5e1e16665eaea43b449bdccc3b3d492\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.011270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.011173 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e1e16665eaea43b449bdccc3b3d492-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"e5e1e16665eaea43b449bdccc3b3d492\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.011270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.011190 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a87cd2889e81c5df0b8771970533f7d1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-63.ec2.internal\" (UID: \"a87cd2889e81c5df0b8771970533f7d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.011270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.011223 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a87cd2889e81c5df0b8771970533f7d1-config\") pod \"kube-apiserver-proxy-ip-10-0-131-63.ec2.internal\" (UID: \"a87cd2889e81c5df0b8771970533f7d1\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.011388 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.011271 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e5e1e16665eaea43b449bdccc3b3d492-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"e5e1e16665eaea43b449bdccc3b3d492\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.011388 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.011279 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e5e1e16665eaea43b449bdccc3b3d492-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal\" (UID: \"e5e1e16665eaea43b449bdccc3b3d492\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.095300 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.095279 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.196047 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.195998 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.221450 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.221429 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.226184 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.226076 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.297084 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.297058 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.397551 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.397524 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.498173 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.498090 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.599280 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.599255 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.638055 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.638026 2586 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 23:29:01.638487 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.638175 2586 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 23:29:01.699463 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:01.699433 2586 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-63.ec2.internal\" not found" Jul 24 23:29:01.726337 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.726296 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 23:24:00 +0000 UTC" deadline="2028-02-15 20:23:37.707563578 +0000 UTC" Jul 24 23:29:01.726337 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.726330 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13700h54m35.981238405s" Jul 24 23:29:01.752499 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.752347 2586 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 23:29:01.809463 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.809439 2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.820093 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.820071 2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 23:29:01.822136 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.822118 2586 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" Jul 24 23:29:01.833982 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.833964 2586 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 23:29:01.857371 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:01.857332 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5e1e16665eaea43b449bdccc3b3d492.slice/crio-4c95976184225d97352ebc37597f434bbbbe00686599489e405ec0b282c3579e WatchSource:0}: Error finding container 4c95976184225d97352ebc37597f434bbbbe00686599489e405ec0b282c3579e: Status 404 returned error can't find the container with id 4c95976184225d97352ebc37597f434bbbbe00686599489e405ec0b282c3579e Jul 24 23:29:01.862420 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.861831 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:29:01.969757 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.969732 2586 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 23:29:01.985233 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:01.985204 2586 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 23:29:02.685785 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.685749 2586 apiserver.go:52] "Watching apiserver" Jul 24 23:29:02.697574 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.697544 2586 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 23:29:02.700663 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.700219 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-zt5mq","openshift-network-diagnostics/network-check-target-lhlft","kube-system/global-pull-secret-syncer-h9j6b","kube-system/konnectivity-agent-tj4jt","openshift-dns/node-resolver-4rjbq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal","openshift-multus/network-metrics-daemon-4s7v8","openshift-network-operator/iptables-alerter-khj8j","openshift-ovn-kubernetes/ovnkube-node-v2t8t","kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv","openshift-cluster-node-tuning-operator/tuned-r9v9w","openshift-image-registry/node-ca-zmcf8","openshift-multus/multus-additional-cni-plugins-vf4nl"] Jul 24 23:29:02.701971 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.701945 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:02.702093 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.702035 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:02.703361 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.703338 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.705304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.705158 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.705304 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.705233 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:02.706366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.706021 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 23:29:02.706366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.706136 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.706366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.706026 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qdqgr\"" Jul 24 23:29:02.706366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.706025 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.707314 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.706770 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.708121 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.708052 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.709449 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.708961 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 23:29:02.709449 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.709296 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xxzm7\"" Jul 24 23:29:02.711051 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.709841 2586 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 23:29:02.711051 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.709860 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 23:29:02.711051 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.710158 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.711051 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.710988 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.713060 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.712205 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.713060 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.712543 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-79lvf\"" Jul 24 23:29:02.713508 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.713488 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.713916 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.713899 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 23:29:02.714235 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.714213 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.714689 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.714669 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 23:29:02.714823 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.714804 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 23:29:02.715474 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.715452 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.716393 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.716368 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.717475 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.717431 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c6sfz\"" Jul 24 23:29:02.722265 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.722244 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h595r\" (UniqueName: \"kubernetes.io/projected/5ff1b64c-69a3-4c3d-846a-3494d996bed5-kube-api-access-h595r\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:02.722396 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.722292 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af3fce95-c009-46a3-81bb-6f841e5c3e6d-konnectivity-ca\") pod \"konnectivity-agent-tj4jt\" (UID: \"af3fce95-c009-46a3-81bb-6f841e5c3e6d\") " pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.722461 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.722391 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:02.722515 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.722494 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:02.722673 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.722651 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-slash\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.723796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723771 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-run-netns\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.723898 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723809 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-systemd\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.723898 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723834 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.723898 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723851 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkzz\" (UniqueName: \"kubernetes.io/projected/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-kube-api-access-wtkzz\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.723898 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723868 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-kubelet\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.723898 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723893 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-systemd-units\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723917 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-etc-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723941 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723966 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723990 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-os-release\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724015 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-dbus\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724039 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-ovn\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724062 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-log-socket\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724090 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724106 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cnibin\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724123 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-kubelet-config\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724139 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-node-log\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724170 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-cni-netd\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724190 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-env-overrides\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724204 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jcmg\" (UniqueName: \"kubernetes.io/projected/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-kube-api-access-8jcmg\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724220 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-host\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724236 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724256 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a863b1e-0369-47ea-81d8-5641fb287599-ovn-node-metrics-cert\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724276 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kg8pv\" (UniqueName: \"kubernetes.io/projected/9a863b1e-0369-47ea-81d8-5641fb287599-kube-api-access-kg8pv\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724295 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpwbr\" (UniqueName: \"kubernetes.io/projected/6ceefe7e-2625-4162-b792-702f2eda86c8-kube-api-access-fpwbr\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724335 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724360 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724433 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724457 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af3fce95-c009-46a3-81bb-6f841e5c3e6d-agent-certs\") pod \"konnectivity-agent-tj4jt\" (UID: \"af3fce95-c009-46a3-81bb-6f841e5c3e6d\") " pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724480 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-var-lib-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724504 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-cni-bin\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724529 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-ovnkube-script-lib\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724553 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ceefe7e-2625-4162-b792-702f2eda86c8-hosts-file\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724575 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.724952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724607 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-serviceca\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724654 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ceefe7e-2625-4162-b792-702f2eda86c8-tmp-dir\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724679 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-system-cni-dir\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724709 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724734 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-ovnkube-config\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723213 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.724859 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723231 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723271 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723278 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723450 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-x89zs\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723497 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723600 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.723639 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6cc4z\"" Jul 24 23:29:02.725819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.725632 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.726799 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.726744 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.727591 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.727337 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.727591 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.727379 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-49kj6\"" Jul 24 23:29:02.727736 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.727608 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 23:29:02.729169 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.729149 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 23:29:02.729323 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.729300 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.729899 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.729879 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.729982 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.729918 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 23:29:02.730178 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.730159 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-96jlv\"" Jul 24 23:29:02.730380 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.730261 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.730380 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.730379 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 23:29:02.730521 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.730158 2586 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 23:29:02.730579 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.730504 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7nt64\"" Jul 24 23:29:02.751200 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.751164 2586 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-tsb4c" Jul 24 23:29:02.759660 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.759490 2586 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tsb4c" Jul 24 23:29:02.786018 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.785404 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" event={"ID":"e5e1e16665eaea43b449bdccc3b3d492","Type":"ContainerStarted","Data":"4c95976184225d97352ebc37597f434bbbbe00686599489e405ec0b282c3579e"} Jul 24 23:29:02.787081 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.787031 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" event={"ID":"a87cd2889e81c5df0b8771970533f7d1","Type":"ContainerStarted","Data":"5549c8162baaf34f13f0b1b78ef5523ed65bf86490b55cfa44a8881f07d46370"} Jul 24 23:29:02.810289 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.810264 2586 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 23:29:02.825558 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.825399 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5th2h\" (UniqueName: \"kubernetes.io/projected/61b42fa0-93a4-4da4-9848-15378da00d8d-kube-api-access-5th2h\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.825558 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.825452 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-host\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.825558 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.825523 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.825838 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.825570 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kg8pv\" (UniqueName: \"kubernetes.io/projected/9a863b1e-0369-47ea-81d8-5641fb287599-kube-api-access-kg8pv\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.825903 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.825885 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.825966 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:03.325932441 +0000 UTC m=+3.018773270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.825590 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpwbr\" (UniqueName: \"kubernetes.io/projected/6ceefe7e-2625-4162-b792-702f2eda86c8-kube-api-access-fpwbr\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826126 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826163 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826338 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826377 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-hostroot\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826411 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-var-lib-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826441 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-lib-modules\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826473 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ece7238a-a34f-4070-a2e1-a01aa69eeab9-tmp\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826509 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd27aa4f-14ce-4f2c-9662-cacfc643f004-cni-binary-copy\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826542 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-serviceca\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826571 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ceefe7e-2625-4162-b792-702f2eda86c8-tmp-dir\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826602 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-system-cni-dir\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826657 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysconfig\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826686 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-kubernetes\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.829182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826714 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-var-lib-kubelet\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826739 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-tuned\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826770 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-cni-bin\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826801 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-daemon-config\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826835 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af3fce95-c009-46a3-81bb-6f841e5c3e6d-konnectivity-ca\") pod \"konnectivity-agent-tj4jt\" (UID: \"af3fce95-c009-46a3-81bb-6f841e5c3e6d\") " pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826871 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-slash\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826897 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-run-netns\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826961 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-registration-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826990 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-sys-fs\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827022 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-socket-dir-parent\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827052 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-conf-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827077 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-etc-kubernetes\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827110 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkzz\" (UniqueName: \"kubernetes.io/projected/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-kube-api-access-wtkzz\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827142 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-kubelet\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827171 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ceefe7e-2625-4162-b792-702f2eda86c8-tmp-dir\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827174 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827245 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-var-lib-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827262 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827252 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-systemd\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827337 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-system-cni-dir\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827376 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jf2z\" (UniqueName: \"kubernetes.io/projected/ece7238a-a34f-4070-a2e1-a01aa69eeab9-kube-api-access-7jf2z\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827405 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-dbus\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827437 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-ovn\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827468 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-log-socket\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827502 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827534 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827564 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-socket-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827596 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysctl-d\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827628 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysctl-conf\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827660 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-kubelet-config\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827692 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-node-log\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827720 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827753 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50200adf-576f-457e-ac42-ea4bd5ba86b6-host-slash\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827783 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-system-cni-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.830987 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827797 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-serviceca\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827813 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-cni-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827850 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-cnibin\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827870 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827883 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-host\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827914 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a863b1e-0369-47ea-81d8-5641fb287599-ovn-node-metrics-cert\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.827955 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.827952 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-device-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828012 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-os-release\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828052 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af3fce95-c009-46a3-81bb-6f841e5c3e6d-agent-certs\") pod \"konnectivity-agent-tj4jt\" (UID: \"af3fce95-c009-46a3-81bb-6f841e5c3e6d\") " pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828132 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-cni-bin\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828169 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-ovnkube-script-lib\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828201 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ceefe7e-2625-4162-b792-702f2eda86c8-hosts-file\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828441 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfk8j\" (UniqueName: \"kubernetes.io/projected/50200adf-576f-457e-ac42-ea4bd5ba86b6-kube-api-access-vfk8j\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828487 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-sys\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828496 2586 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828526 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828557 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-run\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.831612 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828592 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-ovnkube-config\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828597 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/af3fce95-c009-46a3-81bb-6f841e5c3e6d-konnectivity-ca\") pod \"konnectivity-agent-tj4jt\" (UID: \"af3fce95-c009-46a3-81bb-6f841e5c3e6d\") " pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.826832 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cni-binary-copy\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828624 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h595r\" (UniqueName: \"kubernetes.io/projected/5ff1b64c-69a3-4c3d-846a-3494d996bed5-kube-api-access-h595r\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828662 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828685 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-slash\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828699 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-cni-multus\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828737 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-run-netns\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.828746 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:03.328716034 +0000 UTC m=+3.021556860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828773 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6ceefe7e-2625-4162-b792-702f2eda86c8-hosts-file\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828777 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-kubelet\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.828959 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-kubelet\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829165 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-dbus\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829221 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-ovn\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829281 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-log-socket\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829364 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.832566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829638 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-kubelet-config\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829883 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-host\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.829895 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830036 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-node-log\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830117 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-cni-bin\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830485 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-ovnkube-config\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830605 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-ovnkube-script-lib\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830617 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50200adf-576f-457e-ac42-ea4bd5ba86b6-iptables-alerter-script\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830658 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-systemd\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830697 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830732 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-systemd-units\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830766 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-etc-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830798 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830827 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-os-release\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830878 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-k8s-cni-cncf-io\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830911 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cnibin\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830946 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-modprobe-d\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.833383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830993 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-netns\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831021 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-multus-certs\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831054 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9gqg\" (UniqueName: \"kubernetes.io/projected/dd27aa4f-14ce-4f2c-9662-cacfc643f004-kube-api-access-k9gqg\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831092 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-cni-netd\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831132 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-env-overrides\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831161 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jcmg\" (UniqueName: \"kubernetes.io/projected/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-kube-api-access-8jcmg\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831478 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-systemd-units\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831538 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-run-ovn-kubernetes\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.830746 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-run-systemd\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831615 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-host-cni-netd\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831638 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-cnibin\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831692 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-os-release\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.831738 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9a863b1e-0369-47ea-81d8-5641fb287599-etc-openvswitch\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.832021 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.834280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.832048 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9a863b1e-0369-47ea-81d8-5641fb287599-env-overrides\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.835063 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.834382 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/af3fce95-c009-46a3-81bb-6f841e5c3e6d-agent-certs\") pod \"konnectivity-agent-tj4jt\" (UID: \"af3fce95-c009-46a3-81bb-6f841e5c3e6d\") " pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:02.835528 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.835483 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9a863b1e-0369-47ea-81d8-5641fb287599-ovn-node-metrics-cert\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.837294 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.836787 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kg8pv\" (UniqueName: \"kubernetes.io/projected/9a863b1e-0369-47ea-81d8-5641fb287599-kube-api-access-kg8pv\") pod \"ovnkube-node-v2t8t\" (UID: \"9a863b1e-0369-47ea-81d8-5641fb287599\") " pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:02.837294 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.837235 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpwbr\" (UniqueName: \"kubernetes.io/projected/6ceefe7e-2625-4162-b792-702f2eda86c8-kube-api-access-fpwbr\") pod \"node-resolver-4rjbq\" (UID: \"6ceefe7e-2625-4162-b792-702f2eda86c8\") " pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:02.838969 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.838940 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkzz\" (UniqueName: \"kubernetes.io/projected/0afabe2f-09c6-4e92-9bcf-b233c9d64aa0-kube-api-access-wtkzz\") pod \"node-ca-zmcf8\" (UID: \"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0\") " pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:02.839093 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.839072 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h595r\" (UniqueName: \"kubernetes.io/projected/5ff1b64c-69a3-4c3d-846a-3494d996bed5-kube-api-access-h595r\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:02.840407 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.840267 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jcmg\" (UniqueName: \"kubernetes.io/projected/dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e-kube-api-access-8jcmg\") pod \"multus-additional-cni-plugins-vf4nl\" (UID: \"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e\") " pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:02.931977 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.931939 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-modprobe-d\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.931992 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-netns\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932060 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-netns\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932072 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-multus-certs\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932110 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9gqg\" (UniqueName: \"kubernetes.io/projected/dd27aa4f-14ce-4f2c-9662-cacfc643f004-kube-api-access-k9gqg\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932132 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-modprobe-d\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932138 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5th2h\" (UniqueName: \"kubernetes.io/projected/61b42fa0-93a4-4da4-9848-15378da00d8d-kube-api-access-5th2h\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932183 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-host\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932202 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-multus-certs\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932270 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-hostroot\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932287 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-host\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932301 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-lib-modules\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932325 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ece7238a-a34f-4070-a2e1-a01aa69eeab9-tmp\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932334 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-hostroot\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932350 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd27aa4f-14ce-4f2c-9662-cacfc643f004-cni-binary-copy\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932388 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysconfig\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-kubernetes\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932454 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-var-lib-kubelet\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932482 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-lib-modules\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.932494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932487 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-tuned\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932485 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysconfig\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932550 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-kubernetes\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932532 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-var-lib-kubelet\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932619 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-cni-bin\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932670 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-daemon-config\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932707 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-registration-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932721 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-cni-bin\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932736 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-sys-fs\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932773 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-socket-dir-parent\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932795 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-sys-fs\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932802 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-conf-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932919 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-socket-dir-parent\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932956 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-registration-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932967 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-conf-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.932998 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-etc-kubernetes\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933059 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-etc-kubernetes\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933083 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-systemd\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.933126 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933111 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jf2z\" (UniqueName: \"kubernetes.io/projected/ece7238a-a34f-4070-a2e1-a01aa69eeab9-kube-api-access-7jf2z\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933123 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-systemd\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933159 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933178 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-socket-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933203 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysctl-d\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933227 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysctl-conf\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933234 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933268 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933293 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50200adf-576f-457e-ac42-ea4bd5ba86b6-host-slash\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933314 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-system-cni-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933334 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-cni-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933364 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-cnibin\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933365 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-socket-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933366 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysctl-d\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933393 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-device-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933442 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/50200adf-576f-457e-ac42-ea4bd5ba86b6-host-slash\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933444 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-device-dir\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933453 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-system-cni-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933482 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-cnibin\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933489 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-os-release\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933500 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-cni-dir\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933508 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-sysctl-conf\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933561 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-os-release\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933591 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfk8j\" (UniqueName: \"kubernetes.io/projected/50200adf-576f-457e-ac42-ea4bd5ba86b6-kube-api-access-vfk8j\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933635 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-sys\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933674 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-run\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933684 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-sys\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933727 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ece7238a-a34f-4070-a2e1-a01aa69eeab9-run\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933760 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/dd27aa4f-14ce-4f2c-9662-cacfc643f004-cni-binary-copy\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933792 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933820 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-cni-multus\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933846 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-kubelet\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933872 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50200adf-576f-457e-ac42-ea4bd5ba86b6-iptables-alerter-script\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933913 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-k8s-cni-cncf-io\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933923 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-cni-multus\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.934605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933948 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/61b42fa0-93a4-4da4-9848-15378da00d8d-etc-selinux\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:02.935377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933970 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-var-lib-kubelet\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.935377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.933979 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/dd27aa4f-14ce-4f2c-9662-cacfc643f004-host-run-k8s-cni-cncf-io\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.935377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.934039 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/dd27aa4f-14ce-4f2c-9662-cacfc643f004-multus-daemon-config\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.935377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.934556 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/50200adf-576f-457e-ac42-ea4bd5ba86b6-iptables-alerter-script\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.935377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.934921 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ece7238a-a34f-4070-a2e1-a01aa69eeab9-tmp\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.935377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.935096 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ece7238a-a34f-4070-a2e1-a01aa69eeab9-etc-tuned\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.941264 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.941213 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:02.941264 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.941235 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:02.941264 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.941250 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:02.941484 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:02.941342 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:03.441304932 +0000 UTC m=+3.134145755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:02.943752 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.943726 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfk8j\" (UniqueName: \"kubernetes.io/projected/50200adf-576f-457e-ac42-ea4bd5ba86b6-kube-api-access-vfk8j\") pod \"iptables-alerter-khj8j\" (UID: \"50200adf-576f-457e-ac42-ea4bd5ba86b6\") " pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:02.943860 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.943729 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9gqg\" (UniqueName: \"kubernetes.io/projected/dd27aa4f-14ce-4f2c-9662-cacfc643f004-kube-api-access-k9gqg\") pod \"multus-zt5mq\" (UID: \"dd27aa4f-14ce-4f2c-9662-cacfc643f004\") " pod="openshift-multus/multus-zt5mq" Jul 24 23:29:02.944229 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.944207 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jf2z\" (UniqueName: \"kubernetes.io/projected/ece7238a-a34f-4070-a2e1-a01aa69eeab9-kube-api-access-7jf2z\") pod \"tuned-r9v9w\" (UID: \"ece7238a-a34f-4070-a2e1-a01aa69eeab9\") " pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:02.944833 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:02.944812 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5th2h\" (UniqueName: \"kubernetes.io/projected/61b42fa0-93a4-4da4-9848-15378da00d8d-kube-api-access-5th2h\") pod \"aws-ebs-csi-driver-node-z7wrv\" (UID: \"61b42fa0-93a4-4da4-9848-15378da00d8d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:03.027278 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.027228 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-zmcf8" Jul 24 23:29:03.035887 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.035865 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:03.046560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.046539 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4rjbq" Jul 24 23:29:03.052332 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.052310 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" Jul 24 23:29:03.060141 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.060122 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:03.069880 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.069861 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zt5mq" Jul 24 23:29:03.078661 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.078644 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-khj8j" Jul 24 23:29:03.086384 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.086365 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" Jul 24 23:29:03.093833 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.093817 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" Jul 24 23:29:03.229816 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.229733 2586 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 23:29:03.336693 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.336647 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:03.336693 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.336692 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:03.336933 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.336807 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:03.336933 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.336874 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:04.336854241 +0000 UTC m=+4.029695054 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:03.336933 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.336809 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:03.337103 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.336988 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:04.336968781 +0000 UTC m=+4.029809588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:03.537732 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.537641 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:03.537908 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.537823 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:03.537908 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.537847 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:03.537908 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.537858 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:03.538031 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.537924 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:04.537904244 +0000 UTC m=+4.230745076 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:03.761736 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.761685 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 23:24:02 +0000 UTC" deadline="2028-04-24 01:42:40.652032308 +0000 UTC" Jul 24 23:29:03.761736 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.761727 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15338h13m36.890309295s" Jul 24 23:29:03.780269 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:03.780235 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:03.780427 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:03.780402 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:03.931200 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.931168 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd27aa4f_14ce_4f2c_9662_cacfc643f004.slice/crio-cecedf2df72b63aa1ccfb5509b78cf5ca227ef095b3523780b0d0ea60d718c86 WatchSource:0}: Error finding container cecedf2df72b63aa1ccfb5509b78cf5ca227ef095b3523780b0d0ea60d718c86: Status 404 returned error can't find the container with id cecedf2df72b63aa1ccfb5509b78cf5ca227ef095b3523780b0d0ea60d718c86 Jul 24 23:29:03.933278 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.933245 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afabe2f_09c6_4e92_9bcf_b233c9d64aa0.slice/crio-b670b845d37d643ddd4c8520ff70d2fcafaab23e04ba6e94ccbf4d20d21c3f9e WatchSource:0}: Error finding container b670b845d37d643ddd4c8520ff70d2fcafaab23e04ba6e94ccbf4d20d21c3f9e: Status 404 returned error can't find the container with id b670b845d37d643ddd4c8520ff70d2fcafaab23e04ba6e94ccbf4d20d21c3f9e Jul 24 23:29:03.934636 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.934587 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b42fa0_93a4_4da4_9848_15378da00d8d.slice/crio-d58d4b3e0633f8c7ad6334ba2444376c2e050d62d45eb2b266de928173c01b45 WatchSource:0}: Error finding container d58d4b3e0633f8c7ad6334ba2444376c2e050d62d45eb2b266de928173c01b45: Status 404 returned error can't find the container with id d58d4b3e0633f8c7ad6334ba2444376c2e050d62d45eb2b266de928173c01b45 Jul 24 23:29:03.940554 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.940532 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf3fce95_c009_46a3_81bb_6f841e5c3e6d.slice/crio-8190e282327f162ec9f28d6fa9e96f3281300361d1614233163130d54c8e2ebe WatchSource:0}: Error finding container 8190e282327f162ec9f28d6fa9e96f3281300361d1614233163130d54c8e2ebe: Status 404 returned error can't find the container with id 8190e282327f162ec9f28d6fa9e96f3281300361d1614233163130d54c8e2ebe Jul 24 23:29:03.942266 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.942223 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podece7238a_a34f_4070_a2e1_a01aa69eeab9.slice/crio-746f5e726fb13397fbdd5870d22e579d1cf27eb05b67104f91426ee95e764153 WatchSource:0}: Error finding container 746f5e726fb13397fbdd5870d22e579d1cf27eb05b67104f91426ee95e764153: Status 404 returned error can't find the container with id 746f5e726fb13397fbdd5870d22e579d1cf27eb05b67104f91426ee95e764153 Jul 24 23:29:03.944199 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.943934 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a863b1e_0369_47ea_81d8_5641fb287599.slice/crio-d3e224aa4ec4ad63e68514ec3543daa6f09f34869ae3aa32b794ac1f8b5e6291 WatchSource:0}: Error finding container d3e224aa4ec4ad63e68514ec3543daa6f09f34869ae3aa32b794ac1f8b5e6291: Status 404 returned error can't find the container with id d3e224aa4ec4ad63e68514ec3543daa6f09f34869ae3aa32b794ac1f8b5e6291 Jul 24 23:29:03.944969 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.944912 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50200adf_576f_457e_ac42_ea4bd5ba86b6.slice/crio-248fc5d33313a8cde40fd1b7bb3d2570d6a33648fd0c4029628f52a30030c9b9 WatchSource:0}: Error finding container 248fc5d33313a8cde40fd1b7bb3d2570d6a33648fd0c4029628f52a30030c9b9: Status 404 returned error can't find the container with id 248fc5d33313a8cde40fd1b7bb3d2570d6a33648fd0c4029628f52a30030c9b9 Jul 24 23:29:03.946566 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.946545 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8c23b0_c08b_4e85_a58a_e6f4d0bf7e6e.slice/crio-2080b34143732e4b43925756a91a07dbc25ec48969eee3db335618803366de39 WatchSource:0}: Error finding container 2080b34143732e4b43925756a91a07dbc25ec48969eee3db335618803366de39: Status 404 returned error can't find the container with id 2080b34143732e4b43925756a91a07dbc25ec48969eee3db335618803366de39 Jul 24 23:29:03.949203 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:03.948436 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ceefe7e_2625_4162_b792_702f2eda86c8.slice/crio-566082c20bbcf8533cae101dd9da41dbbfac32bb8cac0f5972fd64468ffda035 WatchSource:0}: Error finding container 566082c20bbcf8533cae101dd9da41dbbfac32bb8cac0f5972fd64468ffda035: Status 404 returned error can't find the container with id 566082c20bbcf8533cae101dd9da41dbbfac32bb8cac0f5972fd64468ffda035 Jul 24 23:29:04.343993 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.343775 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:04.344134 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.343954 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:04.344134 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.344069 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:04.344272 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.344147 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:06.344119104 +0000 UTC m=+6.036959917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:04.344272 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.344185 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:04.344272 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.344231 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:06.344215256 +0000 UTC m=+6.037056078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:04.545866 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.545772 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:04.546025 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.545976 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:04.546025 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.546003 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:04.546025 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.546018 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:04.546204 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.546089 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:06.546067774 +0000 UTC m=+6.238908592 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:04.761918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.761846 2586 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 23:24:02 +0000 UTC" deadline="2028-04-30 17:37:19.517975524 +0000 UTC" Jul 24 23:29:04.761918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.761914 2586 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15498h8m14.756065392s" Jul 24 23:29:04.785937 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.785247 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:04.785937 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.785392 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:04.785937 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.785785 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:04.785937 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:04.785876 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:04.814042 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.813841 2586 generic.go:356] "Generic (PLEG): container finished" podID="e5e1e16665eaea43b449bdccc3b3d492" containerID="63cc9adba63fb879daaba21202250e128fea1daa9a77fc103babc5fd9f3450f1" exitCode=0 Jul 24 23:29:04.814042 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.813933 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" event={"ID":"e5e1e16665eaea43b449bdccc3b3d492","Type":"ContainerDied","Data":"63cc9adba63fb879daaba21202250e128fea1daa9a77fc103babc5fd9f3450f1"} Jul 24 23:29:04.818376 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.818347 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4rjbq" event={"ID":"6ceefe7e-2625-4162-b792-702f2eda86c8","Type":"ContainerStarted","Data":"566082c20bbcf8533cae101dd9da41dbbfac32bb8cac0f5972fd64468ffda035"} Jul 24 23:29:04.830078 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.830051 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerStarted","Data":"2080b34143732e4b43925756a91a07dbc25ec48969eee3db335618803366de39"} Jul 24 23:29:04.856634 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.856604 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"d3e224aa4ec4ad63e68514ec3543daa6f09f34869ae3aa32b794ac1f8b5e6291"} Jul 24 23:29:04.866000 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.865973 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khj8j" event={"ID":"50200adf-576f-457e-ac42-ea4bd5ba86b6","Type":"ContainerStarted","Data":"248fc5d33313a8cde40fd1b7bb3d2570d6a33648fd0c4029628f52a30030c9b9"} Jul 24 23:29:04.871385 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.871358 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" event={"ID":"ece7238a-a34f-4070-a2e1-a01aa69eeab9","Type":"ContainerStarted","Data":"746f5e726fb13397fbdd5870d22e579d1cf27eb05b67104f91426ee95e764153"} Jul 24 23:29:04.894257 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.894196 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tj4jt" event={"ID":"af3fce95-c009-46a3-81bb-6f841e5c3e6d","Type":"ContainerStarted","Data":"8190e282327f162ec9f28d6fa9e96f3281300361d1614233163130d54c8e2ebe"} Jul 24 23:29:04.901658 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.901414 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmcf8" event={"ID":"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0","Type":"ContainerStarted","Data":"b670b845d37d643ddd4c8520ff70d2fcafaab23e04ba6e94ccbf4d20d21c3f9e"} Jul 24 23:29:04.910087 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.910059 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" event={"ID":"a87cd2889e81c5df0b8771970533f7d1","Type":"ContainerStarted","Data":"69ace235adbdc5b2be2d3dfa192374e73639b4db450174856051e6453d4dd3ce"} Jul 24 23:29:04.930939 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.930913 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" event={"ID":"61b42fa0-93a4-4da4-9848-15378da00d8d","Type":"ContainerStarted","Data":"d58d4b3e0633f8c7ad6334ba2444376c2e050d62d45eb2b266de928173c01b45"} Jul 24 23:29:04.956992 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:04.956958 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zt5mq" event={"ID":"dd27aa4f-14ce-4f2c-9662-cacfc643f004","Type":"ContainerStarted","Data":"cecedf2df72b63aa1ccfb5509b78cf5ca227ef095b3523780b0d0ea60d718c86"} Jul 24 23:29:05.780823 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:05.780794 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:05.781296 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:05.780931 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:05.980316 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:05.980285 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" event={"ID":"e5e1e16665eaea43b449bdccc3b3d492","Type":"ContainerStarted","Data":"3b0fdbfbe64e0566622d07466762a8fdcdee27e44e26dbce36a832067ab34bd2"} Jul 24 23:29:05.994903 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:05.994851 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-63.ec2.internal" podStartSLOduration=3.994831321 podStartE2EDuration="3.994831321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:04.92654212 +0000 UTC m=+4.619382953" watchObservedRunningTime="2026-07-24 23:29:05.994831321 +0000 UTC m=+5.687672150" Jul 24 23:29:06.360490 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:06.359770 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:06.360490 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:06.359822 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:06.360490 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.360000 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:06.360490 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.360057 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:10.360039139 +0000 UTC m=+10.052879951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:06.360490 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.360125 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:06.360490 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.360158 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:10.360145675 +0000 UTC m=+10.052986489 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:06.561979 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:06.561942 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:06.562152 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.562143 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:06.562211 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.562165 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:06.562211 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.562177 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:06.562337 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.562248 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:10.562214365 +0000 UTC m=+10.255055177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:06.801954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:06.801332 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:06.801954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:06.801370 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:06.801954 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.801470 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:06.801954 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:06.801559 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:07.780737 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:07.780331 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:07.780737 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:07.780435 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:08.780800 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:08.780763 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:08.780800 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:08.780786 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:08.781309 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:08.780965 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:08.781309 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:08.781067 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:09.780531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:09.780452 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:09.780678 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:09.780606 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:10.393585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:10.392949 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:10.393585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:10.392999 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:10.393585 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.393132 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:10.393585 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.393189 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:18.393172105 +0000 UTC m=+18.086012913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:10.394189 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.393618 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:10.394189 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.393701 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:18.393682082 +0000 UTC m=+18.086522894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:10.594443 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:10.594411 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:10.594610 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.594566 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:10.594610 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.594589 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:10.594610 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.594600 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:10.594754 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.594656 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:18.594636712 +0000 UTC m=+18.287477537 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:10.781450 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:10.781373 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:10.781601 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:10.781527 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:10.781657 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.781600 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:10.781657 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:10.781496 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:11.781114 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:11.780629 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:11.781114 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:11.780768 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:12.781955 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:12.781361 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:12.781955 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:12.781472 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:12.781955 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:12.781779 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:12.781955 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:12.781904 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:13.780986 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:13.780951 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:13.781197 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:13.781076 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:14.783833 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:14.783768 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:14.784233 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:14.783768 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:14.784233 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:14.783873 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:14.784233 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:14.783926 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:15.780884 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:15.780851 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:15.781049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:15.780955 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:16.780973 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:16.780930 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:16.780973 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:16.780969 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:16.781553 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:16.781073 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:16.781553 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:16.781226 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:17.781056 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:17.781015 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:17.781597 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:17.781149 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:18.452360 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:18.452274 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:18.452360 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:18.452320 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:18.452567 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.452438 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:18.452567 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.452455 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:18.452567 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.452511 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:34.452489029 +0000 UTC m=+34.145329838 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:18.452567 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.452530 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:34.45252084 +0000 UTC m=+34.145361650 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:18.653012 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:18.652981 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:18.653168 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.653155 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:18.653223 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.653174 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:18.653223 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.653182 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:18.653338 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.653251 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:34.653218893 +0000 UTC m=+34.346059706 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:18.780483 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:18.780407 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:18.780628 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.780530 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:18.780628 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:18.780575 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:18.780730 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:18.780668 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:19.780477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:19.780442 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:19.780908 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:19.780555 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:20.781290 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:20.781255 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:20.781768 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:20.781380 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:20.781768 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:20.781412 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:20.781768 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:20.781593 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:21.780802 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:21.780762 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:21.780989 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:21.780916 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:22.781151 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:22.780849 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:22.781990 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:22.780902 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:22.781990 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:22.781232 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:22.781990 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:22.781303 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:23.016809 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.016719 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" event={"ID":"ece7238a-a34f-4070-a2e1-a01aa69eeab9","Type":"ContainerStarted","Data":"ea0f0bec5e778fd0cb9995532e778a751df540659c4eb741200e9e41138cbc22"} Jul 24 23:29:23.018462 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.018428 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-tj4jt" event={"ID":"af3fce95-c009-46a3-81bb-6f841e5c3e6d","Type":"ContainerStarted","Data":"e7b61d309fbbc7298ab5beea5a340be9073027ab374572da363ae3aaab3f601d"} Jul 24 23:29:23.020581 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.020555 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-zmcf8" event={"ID":"0afabe2f-09c6-4e92-9bcf-b233c9d64aa0","Type":"ContainerStarted","Data":"b6a35b1fdf3bb5c4371cffe675690ee99e62c1dd3518e956431a15bd505ca09f"} Jul 24 23:29:23.022131 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.022106 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" event={"ID":"61b42fa0-93a4-4da4-9848-15378da00d8d","Type":"ContainerStarted","Data":"da860b57d1d8dcbdd5bad024762de44f400e02a023377e0d9d747d50c7915fd1"} Jul 24 23:29:23.023747 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.023724 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zt5mq" event={"ID":"dd27aa4f-14ce-4f2c-9662-cacfc643f004","Type":"ContainerStarted","Data":"c01e11c0b1ee43c1261dde3d55def7ca06ddded3cfb62404595e0e773c0e7b4a"} Jul 24 23:29:23.025260 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.025222 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4rjbq" event={"ID":"6ceefe7e-2625-4162-b792-702f2eda86c8","Type":"ContainerStarted","Data":"926bc93ba9c4054180b039fd437e5cfc6f3f14a9ef939c23fe595bc73f772518"} Jul 24 23:29:23.026801 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.026778 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e" containerID="12322cc56e785f31435f309ae92502971920b315e8c8c7fe4d5573d1038c0c52" exitCode=0 Jul 24 23:29:23.026906 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.026855 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerDied","Data":"12322cc56e785f31435f309ae92502971920b315e8c8c7fe4d5573d1038c0c52"} Jul 24 23:29:23.030210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.030189 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"65ea915343c6e4c3850ad8614c3533303571fb3720121ed04078407b19dcd1d3"} Jul 24 23:29:23.030328 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.030275 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"4e256e7125dbed2c343b4e3536d2c1a0c79edb46dd4c2603f49696fe3e0c4410"} Jul 24 23:29:23.030328 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.030323 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"5ff45e203bc2f0f2165a5b622df1e584baa431882adc4de209cec0d52a798d94"} Jul 24 23:29:23.030432 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.030356 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"f4b7b09c7f46d51994fa8bc2ee2a70d9928fe1473a4f954a160f973cf2d1c806"} Jul 24 23:29:23.030432 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.030393 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"a684e58f77d9e03dde3c9ea1de9c1cc6186d34a7c6cb85e19e98080849a3959f"} Jul 24 23:29:23.030432 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.030430 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"6535a18d1cf0b139bea22d1b2e804fccee003ba855b56320e27024e98e56ff25"} Jul 24 23:29:23.032194 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.032147 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-63.ec2.internal" podStartSLOduration=21.032131468 podStartE2EDuration="21.032131468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:05.995423044 +0000 UTC m=+5.688263876" watchObservedRunningTime="2026-07-24 23:29:23.032131468 +0000 UTC m=+22.724972297" Jul 24 23:29:23.046030 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.045986 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-r9v9w" podStartSLOduration=3.767921413 podStartE2EDuration="22.045972066s" totalImagesPullingTime="18.278050653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:23.031811064 +0000 UTC m=+22.724651892" watchObservedRunningTime="2026-07-24 23:29:23.045972066 +0000 UTC m=+22.738812894" Jul 24 23:29:23.046388 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.046360 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-tj4jt" podStartSLOduration=7.863245669 podStartE2EDuration="22.046350256s" totalImagesPullingTime="14.183104587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:23.045600169 +0000 UTC m=+22.738441000" watchObservedRunningTime="2026-07-24 23:29:23.046350256 +0000 UTC m=+22.739191085" Jul 24 23:29:23.094973 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.094916 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4rjbq" podStartSLOduration=3.824749771 podStartE2EDuration="22.094896373s" totalImagesPullingTime="18.270146602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:23.065488757 +0000 UTC m=+22.758329587" watchObservedRunningTime="2026-07-24 23:29:23.094896373 +0000 UTC m=+22.787737202" Jul 24 23:29:23.136688 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.136638 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-zmcf8" podStartSLOduration=3.851773928 podStartE2EDuration="22.136621045s" totalImagesPullingTime="18.284847117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:23.136491535 +0000 UTC m=+22.829332364" watchObservedRunningTime="2026-07-24 23:29:23.136621045 +0000 UTC m=+22.829461872" Jul 24 23:29:23.136843 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.136823 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zt5mq" podStartSLOduration=3.792544597 podStartE2EDuration="22.136815963s" totalImagesPullingTime="18.344271366s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:23.113981043 +0000 UTC m=+22.806821872" watchObservedRunningTime="2026-07-24 23:29:23.136815963 +0000 UTC m=+22.829656790" Jul 24 23:29:23.780430 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.780402 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:23.780581 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:23.780527 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:23.792695 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:23.792668 2586 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 23:29:24.035045 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.034871 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" event={"ID":"61b42fa0-93a4-4da4-9848-15378da00d8d","Type":"ContainerStarted","Data":"ac6d285bfcfb5ddb62e7e5d9f87c81f2975640d9dc4fa3068e769dee3c3e43db"} Jul 24 23:29:24.036529 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.036498 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-khj8j" event={"ID":"50200adf-576f-457e-ac42-ea4bd5ba86b6","Type":"ContainerStarted","Data":"4c550ec6a3a3536b9208baf735c3f35dea7a47ad65103ae68d005b1426876fb3"} Jul 24 23:29:24.052107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.052061 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-khj8j" podStartSLOduration=4.778300546 podStartE2EDuration="23.052044061s" totalImagesPullingTime="18.273743515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:24.051658583 +0000 UTC m=+23.744499412" watchObservedRunningTime="2026-07-24 23:29:24.052044061 +0000 UTC m=+23.744884926" Jul 24 23:29:24.779077 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.778960 2586 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T23:29:23.792687948Z","UUID":"27a18553-07f6-4723-8a30-fdb9bd653774","Handler":null,"Name":"","Endpoint":""} Jul 24 23:29:24.780196 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.780116 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:24.780196 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.780154 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:24.780405 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:24.780263 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:24.780405 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:24.780350 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:24.780538 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.780522 2586 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 23:29:24.780585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:24.780555 2586 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 23:29:25.780751 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:25.780717 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:25.781190 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:25.780856 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:26.045776 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:26.045564 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"c1be735c55318f27fa26dc2917474a49811212f39410a053d0af90c9105203be"} Jul 24 23:29:26.780813 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:26.780775 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:26.781298 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:26.780788 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:26.781298 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:26.780915 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:26.781298 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:26.781018 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:27.209689 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:27.209657 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:27.210399 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:27.210227 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:27.780586 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:27.780413 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:27.780801 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:27.780689 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:28.051320 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.051184 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e" containerID="732c2c7efdc3b319e652d559df9f78bb8840e6d3049eb102de8292e45dbc6b37" exitCode=0 Jul 24 23:29:28.051320 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.051278 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerDied","Data":"732c2c7efdc3b319e652d559df9f78bb8840e6d3049eb102de8292e45dbc6b37"} Jul 24 23:29:28.055008 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.054985 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" event={"ID":"9a863b1e-0369-47ea-81d8-5641fb287599","Type":"ContainerStarted","Data":"1cc736afda824ce587927d225ad9b0168aff6061a72a7baaf6eeeb10d76a3acf"} Jul 24 23:29:28.055270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.055245 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:28.057258 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.057221 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" event={"ID":"61b42fa0-93a4-4da4-9848-15378da00d8d","Type":"ContainerStarted","Data":"fcc5910a76bd461e1c3756f9ab9dc745898cb01341bdb9dccf3c44f425dad7e3"} Jul 24 23:29:28.057367 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.057349 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:28.057866 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.057853 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-tj4jt" Jul 24 23:29:28.070614 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.070594 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:28.099050 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.098991 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" podStartSLOduration=8.759469082 podStartE2EDuration="27.098976059s" totalImagesPullingTime="18.339506977s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:28.098924478 +0000 UTC m=+27.791765307" watchObservedRunningTime="2026-07-24 23:29:28.098976059 +0000 UTC m=+27.791816887" Jul 24 23:29:28.116117 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.116071 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z7wrv" podStartSLOduration=3.5445631349999998 podStartE2EDuration="27.116055735s" totalImagesPullingTime="23.5714926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:28.115679135 +0000 UTC m=+27.808519963" watchObservedRunningTime="2026-07-24 23:29:28.116055735 +0000 UTC m=+27.808896562" Jul 24 23:29:28.780347 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.780310 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:28.780347 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:28.780333 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:28.780578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:28.780437 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:28.780578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:28.780554 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:29.064739 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.064595 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:29.064739 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.064682 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:29.082855 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.082832 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:29:29.220809 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.220748 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4s7v8"] Jul 24 23:29:29.220996 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.220885 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:29.221067 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:29.220999 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:29.223303 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.223278 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lhlft"] Jul 24 23:29:29.223442 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.223371 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:29.223507 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:29.223464 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:29.231182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.231159 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h9j6b"] Jul 24 23:29:29.231302 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:29.231232 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:29.231347 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:29.231318 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:30.066822 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:30.066787 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e" containerID="ff39bdb17ebb3099f9f118d16a0f5569279d9a73d0d13c25b0be3512af1c782f" exitCode=0 Jul 24 23:29:30.067266 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:30.066875 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerDied","Data":"ff39bdb17ebb3099f9f118d16a0f5569279d9a73d0d13c25b0be3512af1c782f"} Jul 24 23:29:30.781498 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:30.781470 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:30.781656 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:30.781573 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:30.781791 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:30.781762 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:30.781912 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:30.781857 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:30.781912 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:30.781897 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:30.781992 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:30.781960 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:32.073150 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:32.072970 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e" containerID="9cedb900972ec82d3f94682280688ac7e0b107d662e09c97205284bf23dec172" exitCode=0 Jul 24 23:29:32.073547 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:32.073047 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerDied","Data":"9cedb900972ec82d3f94682280688ac7e0b107d662e09c97205284bf23dec172"} Jul 24 23:29:32.781130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:32.781091 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:32.781336 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:32.781144 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:32.781336 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:32.781097 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:32.781336 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:32.781264 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:32.781503 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:32.781417 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:32.781559 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:32.781517 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:34.466903 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:34.466877 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:34.467374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:34.466909 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:34.467374 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.467008 2586 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:34.467374 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.467025 2586 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:34.467374 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.467068 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs podName:5ff1b64c-69a3-4c3d-846a-3494d996bed5 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:06.467051364 +0000 UTC m=+66.159892177 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs") pod "network-metrics-daemon-4s7v8" (UID: "5ff1b64c-69a3-4c3d-846a-3494d996bed5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 23:29:34.467374 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.467085 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret podName:5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:06.467076547 +0000 UTC m=+66.159917358 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret") pod "global-pull-secret-syncer-h9j6b" (UID: "5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52") : object "kube-system"/"original-pull-secret" not registered Jul 24 23:29:34.669519 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:34.669488 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:34.669686 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.669617 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 23:29:34.669686 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.669633 2586 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 23:29:34.669686 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.669642 2586 projected.go:196] Error preparing data for projected volume kube-api-access-jdtpr for pod openshift-network-diagnostics/network-check-target-lhlft: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:34.669814 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.669692 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr podName:27b5f37a-eb1a-4eae-bf0f-394978156f70 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:06.669676528 +0000 UTC m=+66.362517337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-jdtpr" (UniqueName: "kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr") pod "network-check-target-lhlft" (UID: "27b5f37a-eb1a-4eae-bf0f-394978156f70") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 23:29:34.780226 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:34.780202 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:34.780364 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:34.780188 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:34.780364 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:34.780300 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:34.780472 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.780439 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4s7v8" podUID="5ff1b64c-69a3-4c3d-846a-3494d996bed5" Jul 24 23:29:34.780602 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.780558 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lhlft" podUID="27b5f37a-eb1a-4eae-bf0f-394978156f70" Jul 24 23:29:34.780691 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:34.780660 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-h9j6b" podUID="5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52" Jul 24 23:29:35.137863 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.137829 2586 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-63.ec2.internal" event="NodeReady" Jul 24 23:29:35.138028 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.137993 2586 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 23:29:35.175530 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.175498 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64b5b8f786-lj78n"] Jul 24 23:29:35.180460 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.180431 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv"] Jul 24 23:29:35.180588 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.180557 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.183361 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.183339 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bgcmt\"" Jul 24 23:29:35.183479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.183361 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 23:29:35.183479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.183343 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 23:29:35.183479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.183384 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 23:29:35.183882 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.183867 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h"] Jul 24 23:29:35.184288 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.183996 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" Jul 24 23:29:35.186794 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.186775 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.187269 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.187231 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.188465 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.188435 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-btr8c\"" Jul 24 23:29:35.190704 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.189464 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 23:29:35.190704 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.189615 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88"] Jul 24 23:29:35.190704 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.189634 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 23:29:35.190704 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.189689 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.191231 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.191095 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.191231 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.191107 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bj998\"" Jul 24 23:29:35.191231 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.191139 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 23:29:35.191453 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.191438 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.194537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.194116 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg"] Jul 24 23:29:35.194537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.194250 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.196938 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.196741 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 23:29:35.196938 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.196755 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.197088 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.196969 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.197088 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.197071 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jdhgq\"" Jul 24 23:29:35.197191 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.197078 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 23:29:35.198066 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.198049 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv"] Jul 24 23:29:35.198151 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.198075 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b5b8f786-lj78n"] Jul 24 23:29:35.198151 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.198143 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.200388 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.200368 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 23:29:35.202984 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.201632 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.202984 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.202699 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg"] Jul 24 23:29:35.202984 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.202797 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.202984 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.202878 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nvhhl\"" Jul 24 23:29:35.205697 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.205387 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88"] Jul 24 23:29:35.209726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.208862 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mg8tv"] Jul 24 23:29:35.212717 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.212586 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p54z5"] Jul 24 23:29:35.214365 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.214343 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.217208 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.217171 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 23:29:35.217318 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.217213 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qjwqk\"" Jul 24 23:29:35.217503 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.217456 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 23:29:35.218205 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.218147 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.218779 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.218762 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.218981 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.218958 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-59f648499-fvcgf"] Jul 24 23:29:35.219303 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.219260 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.222316 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.222299 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 23:29:35.222560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.222529 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2kgw4\"" Jul 24 23:29:35.223516 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.223503 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 23:29:35.223794 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.223774 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mxztk"] Jul 24 23:29:35.224642 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.224620 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.224839 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.224816 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 23:29:35.229539 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.229325 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9"] Jul 24 23:29:35.229652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.229580 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.230748 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230204 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 23:29:35.230999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230300 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 23:29:35.230999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230359 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 23:29:35.232079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230401 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.232079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230454 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kxkdn\"" Jul 24 23:29:35.232079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230486 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 23:29:35.232079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.230724 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.232321 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.232301 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 23:29:35.232497 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.232478 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.232783 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.232766 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld"] Jul 24 23:29:35.232783 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.232777 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-cqz6m\"" Jul 24 23:29:35.233424 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.233401 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.233699 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.233671 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" Jul 24 23:29:35.235680 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.235660 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.235951 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.235935 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-8lhwc"] Jul 24 23:29:35.236681 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.236664 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.237049 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.237034 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.237177 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.237142 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-mgs5z\"" Jul 24 23:29:35.238908 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.238887 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h"] Jul 24 23:29:35.238998 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.238924 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mxztk"] Jul 24 23:29:35.238998 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.238948 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mg8tv"] Jul 24 23:29:35.238998 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.238949 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.238998 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.238966 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9"] Jul 24 23:29:35.238998 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.238985 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p54z5"] Jul 24 23:29:35.239262 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.239007 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-8lhwc"] Jul 24 23:29:35.239262 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.239023 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59f648499-fvcgf"] Jul 24 23:29:35.239262 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.239026 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.239480 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.239461 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 23:29:35.239562 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.239531 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.240699 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.240683 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kptwj\"" Jul 24 23:29:35.240787 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.240717 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 23:29:35.241306 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.241288 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-2jdnl\"" Jul 24 23:29:35.241378 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.241314 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 23:29:35.241434 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.241418 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 23:29:35.241899 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.241850 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 23:29:35.242519 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.242502 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 23:29:35.246214 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.246197 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld"] Jul 24 23:29:35.248684 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.248666 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 23:29:35.273229 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273202 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v562\" (UniqueName: \"kubernetes.io/projected/2d924791-ad88-44a5-8a8d-90fce288b00a-kube-api-access-2v562\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.273379 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273271 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/957f8e9f-e8fd-4113-9f7c-058c8b1755be-ca-trust-extracted\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.273379 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273317 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvmr\" (UniqueName: \"kubernetes.io/projected/cbc6bcda-8035-4b85-b977-4a4567482781-kube-api-access-shvmr\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.273485 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273375 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/233cbdf6-cc00-4133-8011-6b051a2439df-config\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.273485 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273432 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-certificates\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.273585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273492 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc52s\" (UniqueName: \"kubernetes.io/projected/b43d98b3-bdfd-4517-b631-e4a2292c3b8f-kube-api-access-zc52s\") pod \"volume-data-source-validator-549c6d54cf-p66mv\" (UID: \"b43d98b3-bdfd-4517-b631-e4a2292c3b8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" Jul 24 23:29:35.273585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273521 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b72a9a-44f2-4d63-866b-62cccd430125-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.273585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273565 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-image-registry-private-configuration\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.273730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273589 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-trusted-ca\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.273730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273613 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-bound-sa-token\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.273730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273661 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.273944 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273687 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d924791-ad88-44a5-8a8d-90fce288b00a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.274057 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273963 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b72a9a-44f2-4d63-866b-62cccd430125-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.274057 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.273989 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/233cbdf6-cc00-4133-8011-6b051a2439df-serving-cert\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.274057 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274018 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-installation-pull-secrets\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.274057 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274045 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svp29\" (UniqueName: \"kubernetes.io/projected/e8b72a9a-44f2-4d63-866b-62cccd430125-kube-api-access-svp29\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.274268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274072 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d924791-ad88-44a5-8a8d-90fce288b00a-config\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.274268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274093 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.274268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274115 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233cbdf6-cc00-4133-8011-6b051a2439df-trusted-ca\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.274268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274160 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj64f\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-kube-api-access-rj64f\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.274268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.274198 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2h6r\" (UniqueName: \"kubernetes.io/projected/233cbdf6-cc00-4133-8011-6b051a2439df-kube-api-access-n2h6r\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.346562 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.346510 2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tj4jt_af3fce95-c009-46a3-81bb-6f841e5c3e6d/konnectivity-agent/0.log" limit=1024 Jul 24 23:29:35.374718 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374694 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svp29\" (UniqueName: \"kubernetes.io/projected/e8b72a9a-44f2-4d63-866b-62cccd430125-kube-api-access-svp29\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.374864 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374731 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c76723fb-9238-4421-bc73-5bf67c9604d6-serving-cert\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.374864 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374765 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b72a9a-44f2-4d63-866b-62cccd430125-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.374864 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374798 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-stats-auth\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.374864 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374828 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76723fb-9238-4421-bc73-5bf67c9604d6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.374864 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374853 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb7pd\" (UniqueName: \"kubernetes.io/projected/c76723fb-9238-4421-bc73-5bf67c9604d6-kube-api-access-qb7pd\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.375116 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374906 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233cbdf6-cc00-4133-8011-6b051a2439df-trusted-ca\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.375116 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374949 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj64f\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-kube-api-access-rj64f\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.375116 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374972 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9a7294c-8419-4d34-bb70-554909333f7e-cert\") pod \"ingress-canary-mxztk\" (UID: \"a9a7294c-8419-4d34-bb70-554909333f7e\") " pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.375116 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.374990 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.375116 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375026 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snk9z\" (UniqueName: \"kubernetes.io/projected/a9a7294c-8419-4d34-bb70-554909333f7e-kube-api-access-snk9z\") pod \"ingress-canary-mxztk\" (UID: \"a9a7294c-8419-4d34-bb70-554909333f7e\") " pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375116 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvmr\" (UniqueName: \"kubernetes.io/projected/cbc6bcda-8035-4b85-b977-4a4567482781-kube-api-access-shvmr\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375141 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfz7\" (UniqueName: \"kubernetes.io/projected/32fd7b44-4b9c-4733-a471-3316a037d9f9-kube-api-access-8wfz7\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375166 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76723fb-9238-4421-bc73-5bf67c9604d6-service-ca-bundle\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375195 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/477c6c0e-fc47-4c72-adbe-f697c23204ac-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375225 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b72a9a-44f2-4d63-866b-62cccd430125-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375280 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-image-registry-private-configuration\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375304 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-trusted-ca\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375331 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-bound-sa-token\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375362 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fd7b44-4b9c-4733-a471-3316a037d9f9-tmp-dir\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.375383 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375379 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e8b72a9a-44f2-4d63-866b-62cccd430125-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375399 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375429 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/233cbdf6-cc00-4133-8011-6b051a2439df-serving-cert\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375451 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375475 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d924791-ad88-44a5-8a8d-90fce288b00a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375496 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c76723fb-9238-4421-bc73-5bf67c9604d6-tmp\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375524 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-installation-pull-secrets\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375548 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375572 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d924791-ad88-44a5-8a8d-90fce288b00a-config\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375596 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd7b44-4b9c-4733-a471-3316a037d9f9-config-volume\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375620 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-default-certificate\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375645 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375674 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zv9c\" (UniqueName: \"kubernetes.io/projected/6b862830-5339-43a9-9fed-e990e113ac78-kube-api-access-6zv9c\") pod \"network-check-source-7d88cb55d8-hwpg9\" (UID: \"6b862830-5339-43a9-9fed-e990e113ac78\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375706 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2h6r\" (UniqueName: \"kubernetes.io/projected/233cbdf6-cc00-4133-8011-6b051a2439df-kube-api-access-n2h6r\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375733 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375769 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v562\" (UniqueName: \"kubernetes.io/projected/2d924791-ad88-44a5-8a8d-90fce288b00a-kube-api-access-2v562\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.375867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375796 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/233cbdf6-cc00-4133-8011-6b051a2439df-config\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.375803 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.375820 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5b8f786-lj78n: secret "image-registry-tls" not found Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375821 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlshd\" (UniqueName: \"kubernetes.io/projected/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-kube-api-access-jlshd\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375851 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/957f8e9f-e8fd-4113-9f7c-058c8b1755be-ca-trust-extracted\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.375886 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls podName:957f8e9f-e8fd-4113-9f7c-058c8b1755be nodeName:}" failed. No retries permitted until 2026-07-24 23:29:35.875867029 +0000 UTC m=+35.568707852 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls") pod "image-registry-64b5b8f786-lj78n" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be") : secret "image-registry-tls" not found Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375909 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wgtq\" (UniqueName: \"kubernetes.io/projected/477c6c0e-fc47-4c72-adbe-f697c23204ac-kube-api-access-6wgtq\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375955 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-certificates\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.375987 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc52s\" (UniqueName: \"kubernetes.io/projected/b43d98b3-bdfd-4517-b631-e4a2292c3b8f-kube-api-access-zc52s\") pod \"volume-data-source-validator-549c6d54cf-p66mv\" (UID: \"b43d98b3-bdfd-4517-b631-e4a2292c3b8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.376015 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/233cbdf6-cc00-4133-8011-6b051a2439df-trusted-ca\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.376021 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c76723fb-9238-4421-bc73-5bf67c9604d6-snapshots\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.376181 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/957f8e9f-e8fd-4113-9f7c-058c8b1755be-ca-trust-extracted\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.376344 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-trusted-ca\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.376635 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.376506 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-certificates\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.377230 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.376647 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 23:29:35.377230 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.376706 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls podName:cbc6bcda-8035-4b85-b977-4a4567482781 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:35.876690983 +0000 UTC m=+35.569531808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls") pod "cluster-samples-operator-6748489656-74wqg" (UID: "cbc6bcda-8035-4b85-b977-4a4567482781") : secret "samples-operator-tls" not found Jul 24 23:29:35.377230 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.376707 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2d924791-ad88-44a5-8a8d-90fce288b00a-config\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.377230 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.377170 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/233cbdf6-cc00-4133-8011-6b051a2439df-config\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.379999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.379974 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2d924791-ad88-44a5-8a8d-90fce288b00a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.380323 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.380281 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/233cbdf6-cc00-4133-8011-6b051a2439df-serving-cert\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.380323 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.380294 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-installation-pull-secrets\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.380482 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.380361 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e8b72a9a-44f2-4d63-866b-62cccd430125-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.380534 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.380485 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-image-registry-private-configuration\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.390569 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.390494 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svp29\" (UniqueName: \"kubernetes.io/projected/e8b72a9a-44f2-4d63-866b-62cccd430125-kube-api-access-svp29\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88\" (UID: \"e8b72a9a-44f2-4d63-866b-62cccd430125\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.391856 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.391830 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvmr\" (UniqueName: \"kubernetes.io/projected/cbc6bcda-8035-4b85-b977-4a4567482781-kube-api-access-shvmr\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.391856 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.391850 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-bound-sa-token\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.392037 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.391961 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj64f\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-kube-api-access-rj64f\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.393578 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.393555 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v562\" (UniqueName: \"kubernetes.io/projected/2d924791-ad88-44a5-8a8d-90fce288b00a-kube-api-access-2v562\") pod \"service-ca-operator-5d6c75f84f-9xj5h\" (UID: \"2d924791-ad88-44a5-8a8d-90fce288b00a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.393840 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.393816 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2h6r\" (UniqueName: \"kubernetes.io/projected/233cbdf6-cc00-4133-8011-6b051a2439df-kube-api-access-n2h6r\") pod \"console-operator-584dcb6b54-mg8tv\" (UID: \"233cbdf6-cc00-4133-8011-6b051a2439df\") " pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.393935 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.393856 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc52s\" (UniqueName: \"kubernetes.io/projected/b43d98b3-bdfd-4517-b631-e4a2292c3b8f-kube-api-access-zc52s\") pod \"volume-data-source-validator-549c6d54cf-p66mv\" (UID: \"b43d98b3-bdfd-4517-b631-e4a2292c3b8f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" Jul 24 23:29:35.449345 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.449303 2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tj4jt_af3fce95-c009-46a3-81bb-6f841e5c3e6d/konnectivity-agent/0.log" limit=1024 Jul 24 23:29:35.477294 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477268 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fd7b44-4b9c-4733-a471-3316a037d9f9-tmp-dir\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477330 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477357 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c76723fb-9238-4421-bc73-5bf67c9604d6-tmp\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477394 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd7b44-4b9c-4733-a471-3316a037d9f9-config-volume\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477410 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-default-certificate\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.477469 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.477542 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls podName:32fd7b44-4b9c-4733-a471-3316a037d9f9 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:35.977523084 +0000 UTC m=+35.670363892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls") pod "dns-default-p54z5" (UID: "32fd7b44-4b9c-4733-a471-3316a037d9f9") : secret "dns-default-metrics-tls" not found Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477649 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32fd7b44-4b9c-4733-a471-3316a037d9f9-tmp-dir\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477743 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477761 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c76723fb-9238-4421-bc73-5bf67c9604d6-tmp\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477779 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zv9c\" (UniqueName: \"kubernetes.io/projected/6b862830-5339-43a9-9fed-e990e113ac78-kube-api-access-6zv9c\") pod \"network-check-source-7d88cb55d8-hwpg9\" (UID: \"6b862830-5339-43a9-9fed-e990e113ac78\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477811 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.477843 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477846 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlshd\" (UniqueName: \"kubernetes.io/projected/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-kube-api-access-jlshd\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477874 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wgtq\" (UniqueName: \"kubernetes.io/projected/477c6c0e-fc47-4c72-adbe-f697c23204ac-kube-api-access-6wgtq\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.477891 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls podName:477c6c0e-fc47-4c72-adbe-f697c23204ac nodeName:}" failed. No retries permitted until 2026-07-24 23:29:35.977874119 +0000 UTC m=+35.670714942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-v4jld" (UID: "477c6c0e-fc47-4c72-adbe-f697c23204ac") : secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:35.477970 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477900 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/32fd7b44-4b9c-4733-a471-3316a037d9f9-config-volume\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477929 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c76723fb-9238-4421-bc73-5bf67c9604d6-snapshots\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477961 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c76723fb-9238-4421-bc73-5bf67c9604d6-serving-cert\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.477973 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.477991 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-stats-auth\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478015 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76723fb-9238-4421-bc73-5bf67c9604d6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.478037 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:35.97800994 +0000 UTC m=+35.670850748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : secret "router-metrics-certs-default" not found Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478068 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb7pd\" (UniqueName: \"kubernetes.io/projected/c76723fb-9238-4421-bc73-5bf67c9604d6-kube-api-access-qb7pd\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478117 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9a7294c-8419-4d34-bb70-554909333f7e-cert\") pod \"ingress-canary-mxztk\" (UID: \"a9a7294c-8419-4d34-bb70-554909333f7e\") " pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478174 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478217 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snk9z\" (UniqueName: \"kubernetes.io/projected/a9a7294c-8419-4d34-bb70-554909333f7e-kube-api-access-snk9z\") pod \"ingress-canary-mxztk\" (UID: \"a9a7294c-8419-4d34-bb70-554909333f7e\") " pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478290 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfz7\" (UniqueName: \"kubernetes.io/projected/32fd7b44-4b9c-4733-a471-3316a037d9f9-kube-api-access-8wfz7\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478325 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76723fb-9238-4421-bc73-5bf67c9604d6-service-ca-bundle\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478361 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/477c6c0e-fc47-4c72-adbe-f697c23204ac-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478507 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/c76723fb-9238-4421-bc73-5bf67c9604d6-snapshots\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.478830 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.478806 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:35.978790207 +0000 UTC m=+35.671631027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : configmap references non-existent config key: service-ca.crt Jul 24 23:29:35.479475 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.478963 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76723fb-9238-4421-bc73-5bf67c9604d6-service-ca-bundle\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.479475 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.479108 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/477c6c0e-fc47-4c72-adbe-f697c23204ac-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.479475 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.479163 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c76723fb-9238-4421-bc73-5bf67c9604d6-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.480290 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.480267 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-default-certificate\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.480670 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.480637 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c76723fb-9238-4421-bc73-5bf67c9604d6-serving-cert\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.480806 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.480760 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-stats-auth\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.481481 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.481459 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9a7294c-8419-4d34-bb70-554909333f7e-cert\") pod \"ingress-canary-mxztk\" (UID: \"a9a7294c-8419-4d34-bb70-554909333f7e\") " pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.486167 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.486026 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlshd\" (UniqueName: \"kubernetes.io/projected/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-kube-api-access-jlshd\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.487517 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.487484 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfz7\" (UniqueName: \"kubernetes.io/projected/32fd7b44-4b9c-4733-a471-3316a037d9f9-kube-api-access-8wfz7\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.488077 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.488057 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zv9c\" (UniqueName: \"kubernetes.io/projected/6b862830-5339-43a9-9fed-e990e113ac78-kube-api-access-6zv9c\") pod \"network-check-source-7d88cb55d8-hwpg9\" (UID: \"6b862830-5339-43a9-9fed-e990e113ac78\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" Jul 24 23:29:35.488381 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.488356 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wgtq\" (UniqueName: \"kubernetes.io/projected/477c6c0e-fc47-4c72-adbe-f697c23204ac-kube-api-access-6wgtq\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.488500 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.488476 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-snk9z\" (UniqueName: \"kubernetes.io/projected/a9a7294c-8419-4d34-bb70-554909333f7e-kube-api-access-snk9z\") pod \"ingress-canary-mxztk\" (UID: \"a9a7294c-8419-4d34-bb70-554909333f7e\") " pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.488557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.488509 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb7pd\" (UniqueName: \"kubernetes.io/projected/c76723fb-9238-4421-bc73-5bf67c9604d6-kube-api-access-qb7pd\") pod \"insights-operator-5885fdb759-8lhwc\" (UID: \"c76723fb-9238-4421-bc73-5bf67c9604d6\") " pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.503906 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.503883 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" Jul 24 23:29:35.512742 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.512725 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" Jul 24 23:29:35.521757 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.521732 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" Jul 24 23:29:35.543673 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.543656 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:35.567809 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.567787 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mxztk" Jul 24 23:29:35.575599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.575576 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" Jul 24 23:29:35.587544 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.587522 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" Jul 24 23:29:35.882757 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.882721 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:35.882920 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.882783 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:35.882920 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.882892 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 23:29:35.882920 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.882913 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5b8f786-lj78n: secret "image-registry-tls" not found Jul 24 23:29:35.883049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.882957 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 23:29:35.883049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.882978 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls podName:957f8e9f-e8fd-4113-9f7c-058c8b1755be nodeName:}" failed. No retries permitted until 2026-07-24 23:29:36.882956943 +0000 UTC m=+36.575797778 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls") pod "image-registry-64b5b8f786-lj78n" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be") : secret "image-registry-tls" not found Jul 24 23:29:35.883049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.883003 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls podName:cbc6bcda-8035-4b85-b977-4a4567482781 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:36.882991242 +0000 UTC m=+36.575832051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls") pod "cluster-samples-operator-6748489656-74wqg" (UID: "cbc6bcda-8035-4b85-b977-4a4567482781") : secret "samples-operator-tls" not found Jul 24 23:29:35.983677 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.983648 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:35.983810 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.983706 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:35.983810 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.983753 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.983810 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983803 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983814 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:35.983819 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983866 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls podName:32fd7b44-4b9c-4733-a471-3316a037d9f9 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:36.983847484 +0000 UTC m=+36.676688290 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls") pod "dns-default-p54z5" (UID: "32fd7b44-4b9c-4733-a471-3316a037d9f9") : secret "dns-default-metrics-tls" not found Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983879 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983901 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls podName:477c6c0e-fc47-4c72-adbe-f697c23204ac nodeName:}" failed. No retries permitted until 2026-07-24 23:29:36.983882782 +0000 UTC m=+36.676723603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-v4jld" (UID: "477c6c0e-fc47-4c72-adbe-f697c23204ac") : secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983931 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:36.983913503 +0000 UTC m=+36.676754316 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : secret "router-metrics-certs-default" not found Jul 24 23:29:35.983958 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:35.983957 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:36.983945175 +0000 UTC m=+36.676785988 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : configmap references non-existent config key: service-ca.crt Jul 24 23:29:36.368661 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.368369 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-mg8tv"] Jul 24 23:29:36.371715 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.371672 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88"] Jul 24 23:29:36.372787 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.372744 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9"] Jul 24 23:29:36.374015 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.373932 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-8lhwc"] Jul 24 23:29:36.374829 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.374273 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod233cbdf6_cc00_4133_8011_6b051a2439df.slice/crio-8aab1aedd0b57b7ad02c9cb4c7d2a03ab5c0da68f73455d8e3a337febdeee5a2 WatchSource:0}: Error finding container 8aab1aedd0b57b7ad02c9cb4c7d2a03ab5c0da68f73455d8e3a337febdeee5a2: Status 404 returned error can't find the container with id 8aab1aedd0b57b7ad02c9cb4c7d2a03ab5c0da68f73455d8e3a337febdeee5a2 Jul 24 23:29:36.376282 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.376257 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b72a9a_44f2_4d63_866b_62cccd430125.slice/crio-89bd2d4540f5126c83114e55c305a492c784e0b12ca35c1d081cd468f888149a WatchSource:0}: Error finding container 89bd2d4540f5126c83114e55c305a492c784e0b12ca35c1d081cd468f888149a: Status 404 returned error can't find the container with id 89bd2d4540f5126c83114e55c305a492c784e0b12ca35c1d081cd468f888149a Jul 24 23:29:36.377426 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.377377 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b862830_5339_43a9_9fed_e990e113ac78.slice/crio-cf4a52c84ab6e342e3a98a6cd783af5f7ffb62328dd4ac3dab61206f08dc2ff6 WatchSource:0}: Error finding container cf4a52c84ab6e342e3a98a6cd783af5f7ffb62328dd4ac3dab61206f08dc2ff6: Status 404 returned error can't find the container with id cf4a52c84ab6e342e3a98a6cd783af5f7ffb62328dd4ac3dab61206f08dc2ff6 Jul 24 23:29:36.377811 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.377797 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76723fb_9238_4421_bc73_5bf67c9604d6.slice/crio-7b48fd074a77035abf4595c58a6207b8fc9a243cfdaa7a46d795184849b48823 WatchSource:0}: Error finding container 7b48fd074a77035abf4595c58a6207b8fc9a243cfdaa7a46d795184849b48823: Status 404 returned error can't find the container with id 7b48fd074a77035abf4595c58a6207b8fc9a243cfdaa7a46d795184849b48823 Jul 24 23:29:36.390140 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.385934 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mxztk"] Jul 24 23:29:36.390140 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.388865 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv"] Jul 24 23:29:36.390140 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.389516 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h"] Jul 24 23:29:36.392618 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.392459 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb43d98b3_bdfd_4517_b631_e4a2292c3b8f.slice/crio-10f252481a5f757fa345d28d461c17ef8f95863cbaedfebc07d94e32369cdedc WatchSource:0}: Error finding container 10f252481a5f757fa345d28d461c17ef8f95863cbaedfebc07d94e32369cdedc: Status 404 returned error can't find the container with id 10f252481a5f757fa345d28d461c17ef8f95863cbaedfebc07d94e32369cdedc Jul 24 23:29:36.393349 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.393324 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d924791_ad88_44a5_8a8d_90fce288b00a.slice/crio-636f769f2b11ff42dd5b43eb7079e7de6aa56e702f44c969ea30fd73e1397fa9 WatchSource:0}: Error finding container 636f769f2b11ff42dd5b43eb7079e7de6aa56e702f44c969ea30fd73e1397fa9: Status 404 returned error can't find the container with id 636f769f2b11ff42dd5b43eb7079e7de6aa56e702f44c969ea30fd73e1397fa9 Jul 24 23:29:36.394376 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:36.394355 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a7294c_8419_4d34_bb70_554909333f7e.slice/crio-53fe9a1cc38a6a2f26b32b20485ea198e331c1466b3b9635d6a9d451d753640b WatchSource:0}: Error finding container 53fe9a1cc38a6a2f26b32b20485ea198e331c1466b3b9635d6a9d451d753640b: Status 404 returned error can't find the container with id 53fe9a1cc38a6a2f26b32b20485ea198e331c1466b3b9635d6a9d451d753640b Jul 24 23:29:36.780059 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.780001 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:29:36.780492 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.780002 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:29:36.780492 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.780179 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:29:36.782700 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.782645 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 23:29:36.782700 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.782693 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 23:29:36.782884 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.782776 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t27k5\"" Jul 24 23:29:36.783033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.782984 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nzdt8\"" Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.895999 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.896055 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.896336 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.896351 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5b8f786-lj78n: secret "image-registry-tls" not found Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.896374 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.896405 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls podName:957f8e9f-e8fd-4113-9f7c-058c8b1755be nodeName:}" failed. No retries permitted until 2026-07-24 23:29:38.896389672 +0000 UTC m=+38.589230482 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls") pod "image-registry-64b5b8f786-lj78n" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be") : secret "image-registry-tls" not found Jul 24 23:29:36.896764 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.896434 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls podName:cbc6bcda-8035-4b85-b977-4a4567482781 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:38.896421171 +0000 UTC m=+38.589261977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls") pod "cluster-samples-operator-6748489656-74wqg" (UID: "cbc6bcda-8035-4b85-b977-4a4567482781") : secret "samples-operator-tls" not found Jul 24 23:29:36.997459 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.997394 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:36.997606 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.997533 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:36.997606 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.997597 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:36.997707 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:36.997634 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:36.997792 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.997773 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 23:29:36.997869 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.997857 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:38.997839332 +0000 UTC m=+38.690680151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : secret "router-metrics-certs-default" not found Jul 24 23:29:36.998513 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.998260 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 23:29:36.998513 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.998308 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls podName:32fd7b44-4b9c-4733-a471-3316a037d9f9 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:38.998294252 +0000 UTC m=+38.691135063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls") pod "dns-default-p54z5" (UID: "32fd7b44-4b9c-4733-a471-3316a037d9f9") : secret "dns-default-metrics-tls" not found Jul 24 23:29:36.998513 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.998379 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:38.998368348 +0000 UTC m=+38.691209154 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : configmap references non-existent config key: service-ca.crt Jul 24 23:29:36.998513 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.998439 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:36.998513 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:36.998469 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls podName:477c6c0e-fc47-4c72-adbe-f697c23204ac nodeName:}" failed. No retries permitted until 2026-07-24 23:29:38.998459537 +0000 UTC m=+38.691300358 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-v4jld" (UID: "477c6c0e-fc47-4c72-adbe-f697c23204ac") : secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:37.097422 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.097345 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" event={"ID":"233cbdf6-cc00-4133-8011-6b051a2439df","Type":"ContainerStarted","Data":"8aab1aedd0b57b7ad02c9cb4c7d2a03ab5c0da68f73455d8e3a337febdeee5a2"} Jul 24 23:29:37.099965 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.099935 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" event={"ID":"c76723fb-9238-4421-bc73-5bf67c9604d6","Type":"ContainerStarted","Data":"7b48fd074a77035abf4595c58a6207b8fc9a243cfdaa7a46d795184849b48823"} Jul 24 23:29:37.102345 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.102273 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" event={"ID":"b43d98b3-bdfd-4517-b631-e4a2292c3b8f","Type":"ContainerStarted","Data":"10f252481a5f757fa345d28d461c17ef8f95863cbaedfebc07d94e32369cdedc"} Jul 24 23:29:37.104310 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.104269 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" event={"ID":"2d924791-ad88-44a5-8a8d-90fce288b00a","Type":"ContainerStarted","Data":"636f769f2b11ff42dd5b43eb7079e7de6aa56e702f44c969ea30fd73e1397fa9"} Jul 24 23:29:37.107759 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.107713 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" event={"ID":"6b862830-5339-43a9-9fed-e990e113ac78","Type":"ContainerStarted","Data":"cf4a52c84ab6e342e3a98a6cd783af5f7ffb62328dd4ac3dab61206f08dc2ff6"} Jul 24 23:29:37.111181 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.111117 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" event={"ID":"e8b72a9a-44f2-4d63-866b-62cccd430125","Type":"ContainerStarted","Data":"89bd2d4540f5126c83114e55c305a492c784e0b12ca35c1d081cd468f888149a"} Jul 24 23:29:37.123448 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.122383 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e" containerID="ebd48d224aa7732cc22be1f68be956ddfcc2a5cbd73b8aab385c569b4f3ce277" exitCode=0 Jul 24 23:29:37.123448 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.122555 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerDied","Data":"ebd48d224aa7732cc22be1f68be956ddfcc2a5cbd73b8aab385c569b4f3ce277"} Jul 24 23:29:37.130804 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:37.130744 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mxztk" event={"ID":"a9a7294c-8419-4d34-bb70-554909333f7e","Type":"ContainerStarted","Data":"53fe9a1cc38a6a2f26b32b20485ea198e331c1466b3b9635d6a9d451d753640b"} Jul 24 23:29:38.147070 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:38.146734 2586 generic.go:356] "Generic (PLEG): container finished" podID="dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e" containerID="3c1e3f92280ee58cff3aa6e3726a371875bf39a51099f473a2854044c1db2924" exitCode=0 Jul 24 23:29:38.147070 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:38.146819 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerDied","Data":"3c1e3f92280ee58cff3aa6e3726a371875bf39a51099f473a2854044c1db2924"} Jul 24 23:29:38.919312 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:38.919087 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:38.919476 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:38.919346 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:38.919476 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:38.919265 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 23:29:38.919476 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:38.919394 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5b8f786-lj78n: secret "image-registry-tls" not found Jul 24 23:29:38.919476 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:38.919458 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls podName:957f8e9f-e8fd-4113-9f7c-058c8b1755be nodeName:}" failed. No retries permitted until 2026-07-24 23:29:42.919438638 +0000 UTC m=+42.612279461 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls") pod "image-registry-64b5b8f786-lj78n" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be") : secret "image-registry-tls" not found Jul 24 23:29:38.919655 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:38.919519 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 23:29:38.919655 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:38.919559 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls podName:cbc6bcda-8035-4b85-b977-4a4567482781 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:42.919546298 +0000 UTC m=+42.612387108 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls") pod "cluster-samples-operator-6748489656-74wqg" (UID: "cbc6bcda-8035-4b85-b977-4a4567482781") : secret "samples-operator-tls" not found Jul 24 23:29:39.020465 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:39.020427 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:39.020645 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:39.020546 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:39.020645 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.020616 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:43.020591794 +0000 UTC m=+42.713432603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : configmap references non-existent config key: service-ca.crt Jul 24 23:29:39.020645 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:39.020594 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:39.020834 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:39.020691 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:39.020834 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.020825 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 23:29:39.020938 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.020876 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:43.020860914 +0000 UTC m=+42.713701733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : secret "router-metrics-certs-default" not found Jul 24 23:29:39.020938 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.020937 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 23:29:39.021046 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.020972 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls podName:32fd7b44-4b9c-4733-a471-3316a037d9f9 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:43.020959573 +0000 UTC m=+42.713800395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls") pod "dns-default-p54z5" (UID: "32fd7b44-4b9c-4733-a471-3316a037d9f9") : secret "dns-default-metrics-tls" not found Jul 24 23:29:39.021046 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.021022 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:39.021155 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:39.021084 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls podName:477c6c0e-fc47-4c72-adbe-f697c23204ac nodeName:}" failed. No retries permitted until 2026-07-24 23:29:43.021072562 +0000 UTC m=+42.713913371 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-v4jld" (UID: "477c6c0e-fc47-4c72-adbe-f697c23204ac") : secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:40.462442 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:40.462353 2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tj4jt_af3fce95-c009-46a3-81bb-6f841e5c3e6d/konnectivity-agent/0.log" limit=1024 Jul 24 23:29:40.508268 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:40.508169 2586 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-tj4jt_af3fce95-c009-46a3-81bb-6f841e5c3e6d/konnectivity-agent/0.log" limit=1024 Jul 24 23:29:42.813835 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.813797 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-dbg74"] Jul 24 23:29:42.835569 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.835539 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-dbg74"] Jul 24 23:29:42.835722 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.835635 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:42.838027 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.838001 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 23:29:42.838168 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.838001 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-nslnk\"" Jul 24 23:29:42.838168 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.838005 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 23:29:42.963059 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.963025 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:42.963219 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.963103 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:42.963219 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.963131 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:42.963219 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:42.963187 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4087967-601a-41de-b506-9a66be19ce38-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:42.963379 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:42.963271 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 23:29:42.963379 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:42.963289 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5b8f786-lj78n: secret "image-registry-tls" not found Jul 24 23:29:42.963379 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:42.963316 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 23:29:42.963379 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:42.963361 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls podName:957f8e9f-e8fd-4113-9f7c-058c8b1755be nodeName:}" failed. No retries permitted until 2026-07-24 23:29:50.963342906 +0000 UTC m=+50.656183725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls") pod "image-registry-64b5b8f786-lj78n" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be") : secret "image-registry-tls" not found Jul 24 23:29:42.963379 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:42.963375 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls podName:cbc6bcda-8035-4b85-b977-4a4567482781 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:50.963368418 +0000 UTC m=+50.656209223 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls") pod "cluster-samples-operator-6748489656-74wqg" (UID: "cbc6bcda-8035-4b85-b977-4a4567482781") : secret "samples-operator-tls" not found Jul 24 23:29:43.064630 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.064543 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:43.064630 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.064601 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:43.064630 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.064633 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.064677 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4087967-601a-41de-b506-9a66be19ce38-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064721 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.064742 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.064775 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064797 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls podName:32fd7b44-4b9c-4733-a471-3316a037d9f9 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:51.064775189 +0000 UTC m=+50.757616017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls") pod "dns-default-p54z5" (UID: "32fd7b44-4b9c-4733-a471-3316a037d9f9") : secret "dns-default-metrics-tls" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064799 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064726 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064869 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:51.064852786 +0000 UTC m=+50.757693611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : secret "router-metrics-certs-default" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064875 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064887 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:51.064878023 +0000 UTC m=+50.757718841 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : configmap references non-existent config key: service-ca.crt Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064917 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert podName:f4087967-601a-41de-b506-9a66be19ce38 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:43.56490887 +0000 UTC m=+43.257749676 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dbg74" (UID: "f4087967-601a-41de-b506-9a66be19ce38") : secret "networking-console-plugin-cert" not found Jul 24 23:29:43.065049 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.064945 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls podName:477c6c0e-fc47-4c72-adbe-f697c23204ac nodeName:}" failed. No retries permitted until 2026-07-24 23:29:51.064923695 +0000 UTC m=+50.757764505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-v4jld" (UID: "477c6c0e-fc47-4c72-adbe-f697c23204ac") : secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:43.065683 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.065405 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f4087967-601a-41de-b506-9a66be19ce38-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:43.570270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:43.570217 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:43.570466 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.570393 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 23:29:43.570540 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:43.570472 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert podName:f4087967-601a-41de-b506-9a66be19ce38 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:44.570451223 +0000 UTC m=+44.263292049 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dbg74" (UID: "f4087967-601a-41de-b506-9a66be19ce38") : secret "networking-console-plugin-cert" not found Jul 24 23:29:44.581528 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:44.581478 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:44.582064 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:44.581660 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 23:29:44.582064 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:44.581755 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert podName:f4087967-601a-41de-b506-9a66be19ce38 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:46.581730134 +0000 UTC m=+46.274570959 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dbg74" (UID: "f4087967-601a-41de-b506-9a66be19ce38") : secret "networking-console-plugin-cert" not found Jul 24 23:29:45.169523 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.169486 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" event={"ID":"b43d98b3-bdfd-4517-b631-e4a2292c3b8f","Type":"ContainerStarted","Data":"a7856e929053ce19968209e6fd8d5c36299d4c4d404dbc42ebde4b21970f79ba"} Jul 24 23:29:45.171212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.171187 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" event={"ID":"2d924791-ad88-44a5-8a8d-90fce288b00a","Type":"ContainerStarted","Data":"35fc32e419a23465132d14778d0edc8cb62d7641d787ad37a712db413f5f2739"} Jul 24 23:29:45.172727 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.172694 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" event={"ID":"6b862830-5339-43a9-9fed-e990e113ac78","Type":"ContainerStarted","Data":"e559e59297171e44bc8009edf8cd8f46eec2ea6cee6619a4aae171efe746ff1f"} Jul 24 23:29:45.174274 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.174229 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" event={"ID":"e8b72a9a-44f2-4d63-866b-62cccd430125","Type":"ContainerStarted","Data":"021fabd147e1cef1e3ee0d0548c817b34ab8201dc854e82e8162d70dd38eb635"} Jul 24 23:29:45.177664 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.177643 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" event={"ID":"dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e","Type":"ContainerStarted","Data":"c4aa385c70b1b96e0f631ce8785ca22f4a3cfa45bc2dac07c05eda73a6daa330"} Jul 24 23:29:45.179657 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.179632 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mxztk" event={"ID":"a9a7294c-8419-4d34-bb70-554909333f7e","Type":"ContainerStarted","Data":"43b9d5eb728112ceda0758b27e8f435f7f1dcf053642d812336ab374892e29a0"} Jul 24 23:29:45.181438 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.181418 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/0.log" Jul 24 23:29:45.181520 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.181457 2586 generic.go:356] "Generic (PLEG): container finished" podID="233cbdf6-cc00-4133-8011-6b051a2439df" containerID="7e0a29ee7774e0fe9a1f43be2a02ca3d9a0bce72bd8fb5e5d37183b27310c226" exitCode=255 Jul 24 23:29:45.181579 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.181529 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" event={"ID":"233cbdf6-cc00-4133-8011-6b051a2439df","Type":"ContainerDied","Data":"7e0a29ee7774e0fe9a1f43be2a02ca3d9a0bce72bd8fb5e5d37183b27310c226"} Jul 24 23:29:45.181707 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.181690 2586 scope.go:117] "RemoveContainer" containerID="7e0a29ee7774e0fe9a1f43be2a02ca3d9a0bce72bd8fb5e5d37183b27310c226" Jul 24 23:29:45.183179 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.183156 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" event={"ID":"c76723fb-9238-4421-bc73-5bf67c9604d6","Type":"ContainerStarted","Data":"9a0dcaf96816599649d140d6f8cbb15c39057c09c9bef1cde7da30fa06fb30f5"} Jul 24 23:29:45.186053 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.186007 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-p66mv" podStartSLOduration=15.481527362 podStartE2EDuration="23.185990508s" totalImagesPullingTime="7.704463146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.183579796 +0000 UTC m=+44.876420648" watchObservedRunningTime="2026-07-24 23:29:45.185990508 +0000 UTC m=+44.878831337" Jul 24 23:29:45.222328 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.222267 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" podStartSLOduration=14.498717874 podStartE2EDuration="22.222223645s" totalImagesPullingTime="7.723505771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.221531504 +0000 UTC m=+44.914372335" watchObservedRunningTime="2026-07-24 23:29:45.222223645 +0000 UTC m=+44.915064474" Jul 24 23:29:45.243343 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.242927 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" podStartSLOduration=15.535775613 podStartE2EDuration="23.242906684s" totalImagesPullingTime="7.707131071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.242548883 +0000 UTC m=+44.935389711" watchObservedRunningTime="2026-07-24 23:29:45.242906684 +0000 UTC m=+44.935747513" Jul 24 23:29:45.257737 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.257682 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" podStartSLOduration=15.529793472 podStartE2EDuration="23.257667669s" totalImagesPullingTime="7.727874197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.257510306 +0000 UTC m=+44.950351135" watchObservedRunningTime="2026-07-24 23:29:45.257667669 +0000 UTC m=+44.950508499" Jul 24 23:29:45.274998 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.274921 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mxztk" podStartSLOduration=2.568393241 podStartE2EDuration="10.274899491s" totalImagesPullingTime="7.70650625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.274588965 +0000 UTC m=+44.967429796" watchObservedRunningTime="2026-07-24 23:29:45.274899491 +0000 UTC m=+44.967740323" Jul 24 23:29:45.300767 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.299477 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-hwpg9" podStartSLOduration=14.318279241 podStartE2EDuration="22.299460128s" totalImagesPullingTime="7.981180887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.299404755 +0000 UTC m=+44.992245583" watchObservedRunningTime="2026-07-24 23:29:45.299460128 +0000 UTC m=+44.992300957" Jul 24 23:29:45.311442 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.311403 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8m8fd"] Jul 24 23:29:45.326845 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.325024 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vf4nl" podStartSLOduration=15.82544613 podStartE2EDuration="44.325004426s" totalImagesPullingTime="28.499558296s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:45.324262033 +0000 UTC m=+45.017102860" watchObservedRunningTime="2026-07-24 23:29:45.325004426 +0000 UTC m=+45.017845257" Jul 24 23:29:45.334881 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.334840 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8m8fd"] Jul 24 23:29:45.335198 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.335180 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.338948 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.338928 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 23:29:45.339446 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.339431 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gkx27\"" Jul 24 23:29:45.339844 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.339809 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 23:29:45.499508 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.499416 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.499694 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.499557 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.499694 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.499592 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/daba6781-d561-4345-a105-3f0f93979fe5-crio-socket\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.499694 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.499613 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wl6c\" (UniqueName: \"kubernetes.io/projected/daba6781-d561-4345-a105-3f0f93979fe5-kube-api-access-4wl6c\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.499694 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.499650 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/daba6781-d561-4345-a105-3f0f93979fe5-data-volume\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.544518 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.544480 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:45.544702 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.544556 2586 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:45.600045 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600007 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600045 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600053 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/daba6781-d561-4345-a105-3f0f93979fe5-crio-socket\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600076 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wl6c\" (UniqueName: \"kubernetes.io/projected/daba6781-d561-4345-a105-3f0f93979fe5-kube-api-access-4wl6c\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:45.600154 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600190 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/daba6781-d561-4345-a105-3f0f93979fe5-data-volume\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:45.600249 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:46.1002119 +0000 UTC m=+45.793052711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : configmap "kube-rbac-proxy" not found Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600280 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/daba6781-d561-4345-a105-3f0f93979fe5-crio-socket\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600352 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:45.600518 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.600541 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/daba6781-d561-4345-a105-3f0f93979fe5-data-volume\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:45.600600 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:45.600563 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:46.10054856 +0000 UTC m=+45.793389374 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : secret "insights-runtime-extractor-tls" not found Jul 24 23:29:45.614825 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:45.614796 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wl6c\" (UniqueName: \"kubernetes.io/projected/daba6781-d561-4345-a105-3f0f93979fe5-kube-api-access-4wl6c\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:46.105055 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.105011 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:46.105232 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.105160 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 23:29:46.105232 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.105217 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 23:29:46.105232 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.105224 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:47.10520788 +0000 UTC m=+46.798048686 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : secret "insights-runtime-extractor-tls" not found Jul 24 23:29:46.105232 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.105161 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:46.105409 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.105290 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:47.105273896 +0000 UTC m=+46.798114704 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : configmap "kube-rbac-proxy" not found Jul 24 23:29:46.188285 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.188253 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:29:46.188671 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.188657 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/0.log" Jul 24 23:29:46.188742 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.188692 2586 generic.go:356] "Generic (PLEG): container finished" podID="233cbdf6-cc00-4133-8011-6b051a2439df" containerID="d64b85d8735d8d41e857595acc7bcefbf923c7ac1cd2caebc5a977bbe1a64a87" exitCode=255 Jul 24 23:29:46.188780 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.188739 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" event={"ID":"233cbdf6-cc00-4133-8011-6b051a2439df","Type":"ContainerDied","Data":"d64b85d8735d8d41e857595acc7bcefbf923c7ac1cd2caebc5a977bbe1a64a87"} Jul 24 23:29:46.197806 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.189097 2586 scope.go:117] "RemoveContainer" containerID="7e0a29ee7774e0fe9a1f43be2a02ca3d9a0bce72bd8fb5e5d37183b27310c226" Jul 24 23:29:46.197997 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.197892 2586 scope.go:117] "RemoveContainer" containerID="d64b85d8735d8d41e857595acc7bcefbf923c7ac1cd2caebc5a977bbe1a64a87" Jul 24 23:29:46.198278 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.198225 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mg8tv_openshift-console-operator(233cbdf6-cc00-4133-8011-6b051a2439df)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" podUID="233cbdf6-cc00-4133-8011-6b051a2439df" Jul 24 23:29:46.327231 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.327202 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-8j876"] Jul 24 23:29:46.327231 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.327205 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4rjbq_6ceefe7e-2625-4162-b792-702f2eda86c8/dns-node-resolver/0.log" Jul 24 23:29:46.350430 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.350388 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-8j876"] Jul 24 23:29:46.350618 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.350535 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" Jul 24 23:29:46.352926 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.352900 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 24 23:29:46.353061 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.352932 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 24 23:29:46.353061 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.352945 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-xc4wf\"" Jul 24 23:29:46.408270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.408217 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-757r8\" (UniqueName: \"kubernetes.io/projected/b639319f-b24f-48c6-abcc-06235bcd00c7-kube-api-access-757r8\") pod \"migrator-6865c9967-8j876\" (UID: \"b639319f-b24f-48c6-abcc-06235bcd00c7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" Jul 24 23:29:46.509673 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.509626 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-757r8\" (UniqueName: \"kubernetes.io/projected/b639319f-b24f-48c6-abcc-06235bcd00c7-kube-api-access-757r8\") pod \"migrator-6865c9967-8j876\" (UID: \"b639319f-b24f-48c6-abcc-06235bcd00c7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" Jul 24 23:29:46.517378 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.517348 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-757r8\" (UniqueName: \"kubernetes.io/projected/b639319f-b24f-48c6-abcc-06235bcd00c7-kube-api-access-757r8\") pod \"migrator-6865c9967-8j876\" (UID: \"b639319f-b24f-48c6-abcc-06235bcd00c7\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" Jul 24 23:29:46.610941 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.610846 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:46.611345 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.611001 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 23:29:46.611345 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:46.611068 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert podName:f4087967-601a-41de-b506-9a66be19ce38 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:50.611050479 +0000 UTC m=+50.303891287 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dbg74" (UID: "f4087967-601a-41de-b506-9a66be19ce38") : secret "networking-console-plugin-cert" not found Jul 24 23:29:46.661312 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.661278 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" Jul 24 23:29:46.802894 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:46.802858 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-8j876"] Jul 24 23:29:46.805732 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:46.805702 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb639319f_b24f_48c6_abcc_06235bcd00c7.slice/crio-bcb3b5e0f9334cfe4cd8fc3b93cf9377ad431884b55c263675f1dfadbeeb6601 WatchSource:0}: Error finding container bcb3b5e0f9334cfe4cd8fc3b93cf9377ad431884b55c263675f1dfadbeeb6601: Status 404 returned error can't find the container with id bcb3b5e0f9334cfe4cd8fc3b93cf9377ad431884b55c263675f1dfadbeeb6601 Jul 24 23:29:47.114819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:47.114783 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:47.115038 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:47.114886 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:47.115038 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:47.114931 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 23:29:47.115038 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:47.115013 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:49.114992652 +0000 UTC m=+48.807833473 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : configmap "kube-rbac-proxy" not found Jul 24 23:29:47.115038 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:47.115030 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 23:29:47.115367 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:47.115083 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:49.115070549 +0000 UTC m=+48.807911354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : secret "insights-runtime-extractor-tls" not found Jul 24 23:29:47.194019 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:47.193981 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" event={"ID":"b639319f-b24f-48c6-abcc-06235bcd00c7","Type":"ContainerStarted","Data":"bcb3b5e0f9334cfe4cd8fc3b93cf9377ad431884b55c263675f1dfadbeeb6601"} Jul 24 23:29:47.195484 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:47.195460 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:29:47.195870 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:47.195851 2586 scope.go:117] "RemoveContainer" containerID="d64b85d8735d8d41e857595acc7bcefbf923c7ac1cd2caebc5a977bbe1a64a87" Jul 24 23:29:47.196064 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:47.196043 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-mg8tv_openshift-console-operator(233cbdf6-cc00-4133-8011-6b051a2439df)\"" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" podUID="233cbdf6-cc00-4133-8011-6b051a2439df" Jul 24 23:29:47.721120 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:47.721090 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmcf8_0afabe2f-09c6-4e92-9bcf-b233c9d64aa0/node-ca/0.log" Jul 24 23:29:48.120910 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.120887 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mxztk_a9a7294c-8419-4d34-bb70-554909333f7e/serve-healthcheck-canary/0.log" Jul 24 23:29:48.462292 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.462210 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c96np"] Jul 24 23:29:48.480646 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.480624 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c96np"] Jul 24 23:29:48.480769 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.480724 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.483260 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.483200 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 24 23:29:48.483260 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.483206 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 24 23:29:48.483260 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.483261 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 24 23:29:48.484420 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.484402 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 24 23:29:48.484420 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.484414 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cz529\"" Jul 24 23:29:48.527942 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.527916 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/86e529a5-7d54-4228-8723-b7e1fbf12ee8-signing-key\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.528086 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.528072 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/86e529a5-7d54-4228-8723-b7e1fbf12ee8-signing-cabundle\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.528155 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.528141 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gch5z\" (UniqueName: \"kubernetes.io/projected/86e529a5-7d54-4228-8723-b7e1fbf12ee8-kube-api-access-gch5z\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.628511 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.628482 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/86e529a5-7d54-4228-8723-b7e1fbf12ee8-signing-key\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.628650 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.628634 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/86e529a5-7d54-4228-8723-b7e1fbf12ee8-signing-cabundle\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.628703 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.628689 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gch5z\" (UniqueName: \"kubernetes.io/projected/86e529a5-7d54-4228-8723-b7e1fbf12ee8-kube-api-access-gch5z\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.629274 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.629256 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/86e529a5-7d54-4228-8723-b7e1fbf12ee8-signing-cabundle\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.630664 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.630642 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/86e529a5-7d54-4228-8723-b7e1fbf12ee8-signing-key\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.635497 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.635477 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gch5z\" (UniqueName: \"kubernetes.io/projected/86e529a5-7d54-4228-8723-b7e1fbf12ee8-kube-api-access-gch5z\") pod \"service-ca-658d7b7c78-c96np\" (UID: \"86e529a5-7d54-4228-8723-b7e1fbf12ee8\") " pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.790503 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.790455 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-c96np" Jul 24 23:29:48.924172 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:48.924139 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-c96np"] Jul 24 23:29:48.927650 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:48.927621 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86e529a5_7d54_4228_8723_b7e1fbf12ee8.slice/crio-785c3e90118c64ac46c352801166658048e5b29be220a3c22fde08fdcba6d472 WatchSource:0}: Error finding container 785c3e90118c64ac46c352801166658048e5b29be220a3c22fde08fdcba6d472: Status 404 returned error can't find the container with id 785c3e90118c64ac46c352801166658048e5b29be220a3c22fde08fdcba6d472 Jul 24 23:29:49.133193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.133169 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:49.133315 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.133280 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:49.133315 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:49.133301 2586 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 23:29:49.133386 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:49.133363 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:53.133345423 +0000 UTC m=+52.826186229 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : configmap "kube-rbac-proxy" not found Jul 24 23:29:49.133459 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:49.133429 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 23:29:49.133517 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:49.133479 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:53.133465234 +0000 UTC m=+52.826306054 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : secret "insights-runtime-extractor-tls" not found Jul 24 23:29:49.203153 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.203123 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-c96np" event={"ID":"86e529a5-7d54-4228-8723-b7e1fbf12ee8","Type":"ContainerStarted","Data":"7cf1b76351bbe4b3390e4323e65096eece702f948ecee4847b9f93466c9ddf4f"} Jul 24 23:29:49.203305 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.203179 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-c96np" event={"ID":"86e529a5-7d54-4228-8723-b7e1fbf12ee8","Type":"ContainerStarted","Data":"785c3e90118c64ac46c352801166658048e5b29be220a3c22fde08fdcba6d472"} Jul 24 23:29:49.204897 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.204868 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" event={"ID":"b639319f-b24f-48c6-abcc-06235bcd00c7","Type":"ContainerStarted","Data":"d8f6bc602869c124b90360ab28cdf197b3f19cee534007ff8a9f45d7f7450407"} Jul 24 23:29:49.204996 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.204909 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" event={"ID":"b639319f-b24f-48c6-abcc-06235bcd00c7","Type":"ContainerStarted","Data":"6955614cac7db07c96fab0ec9c2e44b4e9178d98776e95a30db64efe0e7e5fb1"} Jul 24 23:29:49.220887 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.220839 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-c96np" podStartSLOduration=1.220819165 podStartE2EDuration="1.220819165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:49.219693855 +0000 UTC m=+48.912534682" watchObservedRunningTime="2026-07-24 23:29:49.220819165 +0000 UTC m=+48.913659996" Jul 24 23:29:49.239266 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:49.239212 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-8j876" podStartSLOduration=1.5462534209999999 podStartE2EDuration="3.239197426s" totalImagesPullingTime="1.692944005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:49.238167582 +0000 UTC m=+48.931008411" watchObservedRunningTime="2026-07-24 23:29:49.239197426 +0000 UTC m=+48.932038254" Jul 24 23:29:50.646653 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:50.646617 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:50.647107 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:50.646770 2586 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 23:29:50.647107 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:50.646829 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert podName:f4087967-601a-41de-b506-9a66be19ce38 nodeName:}" failed. No retries permitted until 2026-07-24 23:29:58.646814034 +0000 UTC m=+58.339654856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-dbg74" (UID: "f4087967-601a-41de-b506-9a66be19ce38") : secret "networking-console-plugin-cert" not found Jul 24 23:29:51.051363 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:51.051284 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:29:51.051363 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:51.051338 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:29:51.051578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.051433 2586 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 23:29:51.051578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.051452 2586 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64b5b8f786-lj78n: secret "image-registry-tls" not found Jul 24 23:29:51.051578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.051475 2586 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 23:29:51.051578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.051507 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls podName:957f8e9f-e8fd-4113-9f7c-058c8b1755be nodeName:}" failed. No retries permitted until 2026-07-24 23:30:07.051489696 +0000 UTC m=+66.744330526 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls") pod "image-registry-64b5b8f786-lj78n" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be") : secret "image-registry-tls" not found Jul 24 23:29:51.051578 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.051527 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls podName:cbc6bcda-8035-4b85-b977-4a4567482781 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:07.051512668 +0000 UTC m=+66.744353496 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls") pod "cluster-samples-operator-6748489656-74wqg" (UID: "cbc6bcda-8035-4b85-b977-4a4567482781") : secret "samples-operator-tls" not found Jul 24 23:29:51.151845 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:51.151816 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:29:51.151985 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:51.151852 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:51.151985 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:51.151913 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:29:51.151985 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.151934 2586 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:51.151985 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.151987 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls podName:477c6c0e-fc47-4c72-adbe-f697c23204ac nodeName:}" failed. No retries permitted until 2026-07-24 23:30:07.151973471 +0000 UTC m=+66.844814277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-v4jld" (UID: "477c6c0e-fc47-4c72-adbe-f697c23204ac") : secret "cluster-monitoring-operator-tls" not found Jul 24 23:29:51.155660 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.152010 2586 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 23:29:51.155660 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.152413 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:07.152205192 +0000 UTC m=+66.845046015 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : configmap references non-existent config key: service-ca.crt Jul 24 23:29:51.155660 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.152448 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs podName:d363ed1b-691f-4e1e-9416-5a8bb29e0a43 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:07.152429161 +0000 UTC m=+66.845269981 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs") pod "router-default-59f648499-fvcgf" (UID: "d363ed1b-691f-4e1e-9416-5a8bb29e0a43") : secret "router-metrics-certs-default" not found Jul 24 23:29:51.155660 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:51.152636 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:29:51.155660 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.152795 2586 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 23:29:51.155660 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:51.152864 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls podName:32fd7b44-4b9c-4733-a471-3316a037d9f9 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:07.15284922 +0000 UTC m=+66.845690040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls") pod "dns-default-p54z5" (UID: "32fd7b44-4b9c-4733-a471-3316a037d9f9") : secret "dns-default-metrics-tls" not found Jul 24 23:29:53.169753 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:53.169722 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:53.170157 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:53.169871 2586 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 23:29:53.170157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:53.169889 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:53.170157 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:29:53.169942 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls podName:daba6781-d561-4345-a105-3f0f93979fe5 nodeName:}" failed. No retries permitted until 2026-07-24 23:30:01.169926466 +0000 UTC m=+60.862767273 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8m8fd" (UID: "daba6781-d561-4345-a105-3f0f93979fe5") : secret "insights-runtime-extractor-tls" not found Jul 24 23:29:53.170370 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:53.170354 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/daba6781-d561-4345-a105-3f0f93979fe5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:29:55.544163 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:55.544127 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:55.544654 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:55.544192 2586 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:55.544654 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:55.544596 2586 scope.go:117] "RemoveContainer" containerID="d64b85d8735d8d41e857595acc7bcefbf923c7ac1cd2caebc5a977bbe1a64a87" Jul 24 23:29:56.235445 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:56.235416 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:29:56.235605 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:56.235504 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" event={"ID":"233cbdf6-cc00-4133-8011-6b051a2439df","Type":"ContainerStarted","Data":"07550cbbad97d6c7bf2f0ee866bf2858a469675b6f758cb32bf7afc11938325a"} Jul 24 23:29:56.235796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:56.235772 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:56.240578 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:56.240558 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" Jul 24 23:29:56.254882 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:56.254846 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-mg8tv" podStartSLOduration=25.529164937 podStartE2EDuration="33.254834673s" totalImagesPullingTime="7.725669736s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:29:56.25394113 +0000 UTC m=+55.946781960" watchObservedRunningTime="2026-07-24 23:29:56.254834673 +0000 UTC m=+55.947675479" Jul 24 23:29:58.714620 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:58.714589 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:58.716859 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:58.716832 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f4087967-601a-41de-b506-9a66be19ce38-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-dbg74\" (UID: \"f4087967-601a-41de-b506-9a66be19ce38\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:58.744713 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:58.744692 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" Jul 24 23:29:58.865267 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:58.865227 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-dbg74"] Jul 24 23:29:58.867807 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:29:58.867778 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4087967_601a_41de_b506_9a66be19ce38.slice/crio-3bedeb152ffbec2b0ddb643a03d99417ecead6f888c3271a8ffe88bc9e94954f WatchSource:0}: Error finding container 3bedeb152ffbec2b0ddb643a03d99417ecead6f888c3271a8ffe88bc9e94954f: Status 404 returned error can't find the container with id 3bedeb152ffbec2b0ddb643a03d99417ecead6f888c3271a8ffe88bc9e94954f Jul 24 23:29:59.248490 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:29:59.248460 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" event={"ID":"f4087967-601a-41de-b506-9a66be19ce38","Type":"ContainerStarted","Data":"3bedeb152ffbec2b0ddb643a03d99417ecead6f888c3271a8ffe88bc9e94954f"} Jul 24 23:30:01.097171 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.097142 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-v2t8t" Jul 24 23:30:01.236463 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.236432 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:30:01.238654 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.238629 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/daba6781-d561-4345-a105-3f0f93979fe5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8m8fd\" (UID: \"daba6781-d561-4345-a105-3f0f93979fe5\") " pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:30:01.253301 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.253282 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-gkx27\"" Jul 24 23:30:01.258050 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.258025 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" event={"ID":"f4087967-601a-41de-b506-9a66be19ce38","Type":"ContainerStarted","Data":"0f74ebfd0bdab07bd69a3c258894b91ab8318f1a5f95b3dbe65dbdf9913c5bcb"} Jul 24 23:30:01.260866 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.260852 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8m8fd" Jul 24 23:30:01.272537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.272496 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-dbg74" podStartSLOduration=17.681772307 podStartE2EDuration="19.272483086s" totalImagesPullingTime="1.590710779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:01.272229616 +0000 UTC m=+60.965070446" watchObservedRunningTime="2026-07-24 23:30:01.272483086 +0000 UTC m=+60.965323956" Jul 24 23:30:01.410498 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:01.410456 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8m8fd"] Jul 24 23:30:01.413818 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:01.413788 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaba6781_d561_4345_a105_3f0f93979fe5.slice/crio-17b78989aea2e92a969bd02f329e3747f6daaf747e4c11e1551bbe8dd3290480 WatchSource:0}: Error finding container 17b78989aea2e92a969bd02f329e3747f6daaf747e4c11e1551bbe8dd3290480: Status 404 returned error can't find the container with id 17b78989aea2e92a969bd02f329e3747f6daaf747e4c11e1551bbe8dd3290480 Jul 24 23:30:02.262719 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:02.262679 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m8fd" event={"ID":"daba6781-d561-4345-a105-3f0f93979fe5","Type":"ContainerStarted","Data":"17b78989aea2e92a969bd02f329e3747f6daaf747e4c11e1551bbe8dd3290480"} Jul 24 23:30:06.491504 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.491468 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:30:06.491504 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.491505 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:30:06.494654 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.494630 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 23:30:06.494778 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.494630 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 23:30:06.503982 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.503940 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52-original-pull-secret\") pod \"global-pull-secret-syncer-h9j6b\" (UID: \"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52\") " pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:30:06.504089 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.504069 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ff1b64c-69a3-4c3d-846a-3494d996bed5-metrics-certs\") pod \"network-metrics-daemon-4s7v8\" (UID: \"5ff1b64c-69a3-4c3d-846a-3494d996bed5\") " pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:30:06.692848 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.692808 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:30:06.695924 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.695900 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtpr\" (UniqueName: \"kubernetes.io/projected/27b5f37a-eb1a-4eae-bf0f-394978156f70-kube-api-access-jdtpr\") pod \"network-check-target-lhlft\" (UID: \"27b5f37a-eb1a-4eae-bf0f-394978156f70\") " pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:30:06.792776 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.792755 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-t27k5\"" Jul 24 23:30:06.798144 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.798117 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nzdt8\"" Jul 24 23:30:06.799853 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.799835 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-h9j6b" Jul 24 23:30:06.800980 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.800963 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4s7v8" Jul 24 23:30:06.806323 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.806305 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:30:06.953554 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:06.953497 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4s7v8"] Jul 24 23:30:06.955938 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:06.955905 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ff1b64c_69a3_4c3d_846a_3494d996bed5.slice/crio-f5367ee30d9c45d7c2e1dbb2171d4a1343722ac2001cd9cda14abe3daea5a5bb WatchSource:0}: Error finding container f5367ee30d9c45d7c2e1dbb2171d4a1343722ac2001cd9cda14abe3daea5a5bb: Status 404 returned error can't find the container with id f5367ee30d9c45d7c2e1dbb2171d4a1343722ac2001cd9cda14abe3daea5a5bb Jul 24 23:30:07.095780 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.095718 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:07.095780 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.095762 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:30:07.098098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.098063 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/cbc6bcda-8035-4b85-b977-4a4567482781-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-74wqg\" (UID: \"cbc6bcda-8035-4b85-b977-4a4567482781\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:30:07.098098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.098085 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"image-registry-64b5b8f786-lj78n\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:07.172361 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.172330 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lhlft"] Jul 24 23:30:07.173549 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.173526 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-h9j6b"] Jul 24 23:30:07.175326 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:07.175297 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b5f37a_eb1a_4eae_bf0f_394978156f70.slice/crio-34433a8681612a68b99f46828c58ebeeed56632babefab04a2eaf08e0516421d WatchSource:0}: Error finding container 34433a8681612a68b99f46828c58ebeeed56632babefab04a2eaf08e0516421d: Status 404 returned error can't find the container with id 34433a8681612a68b99f46828c58ebeeed56632babefab04a2eaf08e0516421d Jul 24 23:30:07.176316 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:07.176288 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b2c2b1c_c32a_45e8_ac58_b00ff25ffe52.slice/crio-a33aa0272aeb5bb6a4aa088a2ef03278a841418155b8dbeedd1604b61939e859 WatchSource:0}: Error finding container a33aa0272aeb5bb6a4aa088a2ef03278a841418155b8dbeedd1604b61939e859: Status 404 returned error can't find the container with id a33aa0272aeb5bb6a4aa088a2ef03278a841418155b8dbeedd1604b61939e859 Jul 24 23:30:07.196460 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.196419 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:07.196978 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.196713 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:30:07.196978 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.196769 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:30:07.196978 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.196808 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:07.197496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.197475 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-service-ca-bundle\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:07.199481 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.199463 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d363ed1b-691f-4e1e-9416-5a8bb29e0a43-metrics-certs\") pod \"router-default-59f648499-fvcgf\" (UID: \"d363ed1b-691f-4e1e-9416-5a8bb29e0a43\") " pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:07.199568 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.199469 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/477c6c0e-fc47-4c72-adbe-f697c23204ac-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-v4jld\" (UID: \"477c6c0e-fc47-4c72-adbe-f697c23204ac\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:30:07.199651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.199633 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/32fd7b44-4b9c-4733-a471-3316a037d9f9-metrics-tls\") pod \"dns-default-p54z5\" (UID: \"32fd7b44-4b9c-4733-a471-3316a037d9f9\") " pod="openshift-dns/dns-default-p54z5" Jul 24 23:30:07.282305 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.282272 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m8fd" event={"ID":"daba6781-d561-4345-a105-3f0f93979fe5","Type":"ContainerStarted","Data":"24d7827f660525a3e882ed2fe8ef6f115e79c80e279c8f9e22d472f052447fde"} Jul 24 23:30:07.283945 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.283909 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lhlft" event={"ID":"27b5f37a-eb1a-4eae-bf0f-394978156f70","Type":"ContainerStarted","Data":"b95e2e6fb69ebf7ae8214e93388a91ef895f517b048eb52643c3dec6117e52df"} Jul 24 23:30:07.284070 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.283979 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:30:07.284070 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.284000 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lhlft" event={"ID":"27b5f37a-eb1a-4eae-bf0f-394978156f70","Type":"ContainerStarted","Data":"34433a8681612a68b99f46828c58ebeeed56632babefab04a2eaf08e0516421d"} Jul 24 23:30:07.285251 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.285214 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4s7v8" event={"ID":"5ff1b64c-69a3-4c3d-846a-3494d996bed5","Type":"ContainerStarted","Data":"f5367ee30d9c45d7c2e1dbb2171d4a1343722ac2001cd9cda14abe3daea5a5bb"} Jul 24 23:30:07.286417 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.286393 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h9j6b" event={"ID":"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52","Type":"ContainerStarted","Data":"a33aa0272aeb5bb6a4aa088a2ef03278a841418155b8dbeedd1604b61939e859"} Jul 24 23:30:07.296621 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.296600 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bgcmt\"" Jul 24 23:30:07.299544 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.299500 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lhlft" podStartSLOduration=66.299488918 podStartE2EDuration="1m6.299488918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:07.297839805 +0000 UTC m=+66.990680634" watchObservedRunningTime="2026-07-24 23:30:07.299488918 +0000 UTC m=+66.992329747" Jul 24 23:30:07.304947 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.304922 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:07.336060 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.336035 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-nvhhl\"" Jul 24 23:30:07.343279 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.343257 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" Jul 24 23:30:07.354525 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.354505 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2kgw4\"" Jul 24 23:30:07.361983 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.361943 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p54z5" Jul 24 23:30:07.362380 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.362326 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-kxkdn\"" Jul 24 23:30:07.369405 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.369383 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:07.385912 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.385673 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-kptwj\"" Jul 24 23:30:07.392698 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.392644 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" Jul 24 23:30:07.532172 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.531541 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64b5b8f786-lj78n"] Jul 24 23:30:07.551807 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:07.551772 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957f8e9f_e8fd_4113_9f7c_058c8b1755be.slice/crio-082069b1f8cf419c640a69e8f11a49b82a3cb1dcb305b23c05399b789e186dd6 WatchSource:0}: Error finding container 082069b1f8cf419c640a69e8f11a49b82a3cb1dcb305b23c05399b789e186dd6: Status 404 returned error can't find the container with id 082069b1f8cf419c640a69e8f11a49b82a3cb1dcb305b23c05399b789e186dd6 Jul 24 23:30:07.571083 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.570925 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg"] Jul 24 23:30:07.601706 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.601511 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p54z5"] Jul 24 23:30:07.604088 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:07.604033 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32fd7b44_4b9c_4733_a471_3316a037d9f9.slice/crio-83e81ffc503795e106636ee237cbb43ef3d6601524acbeeb37cba1428b382e40 WatchSource:0}: Error finding container 83e81ffc503795e106636ee237cbb43ef3d6601524acbeeb37cba1428b382e40: Status 404 returned error can't find the container with id 83e81ffc503795e106636ee237cbb43ef3d6601524acbeeb37cba1428b382e40 Jul 24 23:30:07.640167 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.640032 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-59f648499-fvcgf"] Jul 24 23:30:07.644452 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:07.644371 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd363ed1b_691f_4e1e_9416_5a8bb29e0a43.slice/crio-2a0d0c25c1821b1ab6365ff7b81f50df5e270bdd9e0f1bb8fa72dc550989f601 WatchSource:0}: Error finding container 2a0d0c25c1821b1ab6365ff7b81f50df5e270bdd9e0f1bb8fa72dc550989f601: Status 404 returned error can't find the container with id 2a0d0c25c1821b1ab6365ff7b81f50df5e270bdd9e0f1bb8fa72dc550989f601 Jul 24 23:30:07.662021 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:07.661996 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld"] Jul 24 23:30:07.665112 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:07.665088 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477c6c0e_fc47_4c72_adbe_f697c23204ac.slice/crio-20316118185aba9ba71842d51ed86e0fca53a694859c0f028951fb7f94f83c52 WatchSource:0}: Error finding container 20316118185aba9ba71842d51ed86e0fca53a694859c0f028951fb7f94f83c52: Status 404 returned error can't find the container with id 20316118185aba9ba71842d51ed86e0fca53a694859c0f028951fb7f94f83c52 Jul 24 23:30:08.225665 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.225493 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl"] Jul 24 23:30:08.252273 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.252225 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-8rpg2"] Jul 24 23:30:08.254534 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.252491 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.259359 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.258959 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 23:30:08.264408 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.264202 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 23:30:08.264535 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.264431 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-8h7gn\"" Jul 24 23:30:08.264869 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.264692 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 23:30:08.264869 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.264783 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 23:30:08.272318 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.272174 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8"] Jul 24 23:30:08.273672 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.273651 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:08.277580 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.277340 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-gk9zz\"" Jul 24 23:30:08.277580 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.277463 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 23:30:08.278487 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.278304 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 23:30:08.283518 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.282872 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm"] Jul 24 23:30:08.283518 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.283466 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.292489 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.292057 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 23:30:08.293221 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.292595 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl"] Jul 24 23:30:08.293221 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.292626 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8"] Jul 24 23:30:08.293221 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.292660 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-8rpg2"] Jul 24 23:30:08.293221 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.292678 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm"] Jul 24 23:30:08.293221 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.292768 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.296334 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.295309 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 23:30:08.296334 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.295817 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 23:30:08.296334 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.295897 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 23:30:08.297679 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.297519 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 23:30:08.300473 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.300444 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59f648499-fvcgf" event={"ID":"d363ed1b-691f-4e1e-9416-5a8bb29e0a43","Type":"ContainerStarted","Data":"7952c4b2dd9573ca65ab9984a26a480f2965cb2828cba3d8bc448de218d65077"} Jul 24 23:30:08.300658 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.300640 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-59f648499-fvcgf" event={"ID":"d363ed1b-691f-4e1e-9416-5a8bb29e0a43","Type":"ContainerStarted","Data":"2a0d0c25c1821b1ab6365ff7b81f50df5e270bdd9e0f1bb8fa72dc550989f601"} Jul 24 23:30:08.310157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.309506 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" event={"ID":"cbc6bcda-8035-4b85-b977-4a4567482781","Type":"ContainerStarted","Data":"7ba14d9b6a4ea9f0b5cb37722688d6cffa8a8c802de4c40a64fdab82b9795729"} Jul 24 23:30:08.318586 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.318526 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p54z5" event={"ID":"32fd7b44-4b9c-4733-a471-3316a037d9f9","Type":"ContainerStarted","Data":"83e81ffc503795e106636ee237cbb43ef3d6601524acbeeb37cba1428b382e40"} Jul 24 23:30:08.328335 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.327609 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" event={"ID":"477c6c0e-fc47-4c72-adbe-f697c23204ac","Type":"ContainerStarted","Data":"20316118185aba9ba71842d51ed86e0fca53a694859c0f028951fb7f94f83c52"} Jul 24 23:30:08.333750 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.333457 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" event={"ID":"957f8e9f-e8fd-4113-9f7c-058c8b1755be","Type":"ContainerStarted","Data":"90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72"} Jul 24 23:30:08.333750 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.333500 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" event={"ID":"957f8e9f-e8fd-4113-9f7c-058c8b1755be","Type":"ContainerStarted","Data":"082069b1f8cf419c640a69e8f11a49b82a3cb1dcb305b23c05399b789e186dd6"} Jul 24 23:30:08.334169 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.334141 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:08.369891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.369556 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" podStartSLOduration=67.369537867 podStartE2EDuration="1m7.369537867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:08.368543529 +0000 UTC m=+68.061384358" watchObservedRunningTime="2026-07-24 23:30:08.369537867 +0000 UTC m=+68.062378699" Jul 24 23:30:08.369891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.369729 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:08.372954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.372915 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.408942 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe10d3b9-2dd4-4f73-8174-d888be2b8edb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8475b48744-ckjgl\" (UID: \"fe10d3b9-2dd4-4f73-8174-d888be2b8edb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.408993 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8qqb\" (UniqueName: \"kubernetes.io/projected/69db2f4a-d983-40c1-950f-2d74d5ba4438-kube-api-access-z8qqb\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409028 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409111 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-hub\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409165 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409311 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rc2p\" (UniqueName: \"kubernetes.io/projected/006316eb-e088-4399-8d77-01501525c79c-kube-api-access-2rc2p\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409366 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/006316eb-e088-4399-8d77-01501525c79c-klusterlet-config\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409435 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/69db2f4a-d983-40c1-950f-2d74d5ba4438-additional-service-ca\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409477 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxmq6\" (UniqueName: \"kubernetes.io/projected/1582bf21-f4b1-4983-940a-76597189c439-kube-api-access-bxmq6\") pod \"downloads-7c97dc4555-8rpg2\" (UID: \"1582bf21-f4b1-4983-940a-76597189c439\") " pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409536 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-ca\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409563 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jffv\" (UniqueName: \"kubernetes.io/projected/fe10d3b9-2dd4-4f73-8174-d888be2b8edb-kube-api-access-6jffv\") pod \"managed-serviceaccount-addon-agent-8475b48744-ckjgl\" (UID: \"fe10d3b9-2dd4-4f73-8174-d888be2b8edb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.411537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.409630 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006316eb-e088-4399-8d77-01501525c79c-tmp\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.420500 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.419602 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-59f648499-fvcgf" podStartSLOduration=45.419583453 podStartE2EDuration="45.419583453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:08.418433934 +0000 UTC m=+68.111274762" watchObservedRunningTime="2026-07-24 23:30:08.419583453 +0000 UTC m=+68.112424283" Jul 24 23:30:08.510429 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510357 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rc2p\" (UniqueName: \"kubernetes.io/projected/006316eb-e088-4399-8d77-01501525c79c-kube-api-access-2rc2p\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.510429 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510408 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/006316eb-e088-4399-8d77-01501525c79c-klusterlet-config\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.510623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510435 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/69db2f4a-d983-40c1-950f-2d74d5ba4438-additional-service-ca\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.510623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510468 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxmq6\" (UniqueName: \"kubernetes.io/projected/1582bf21-f4b1-4983-940a-76597189c439-kube-api-access-bxmq6\") pod \"downloads-7c97dc4555-8rpg2\" (UID: \"1582bf21-f4b1-4983-940a-76597189c439\") " pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:08.510623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510505 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-ca\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.510623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510530 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jffv\" (UniqueName: \"kubernetes.io/projected/fe10d3b9-2dd4-4f73-8174-d888be2b8edb-kube-api-access-6jffv\") pod \"managed-serviceaccount-addon-agent-8475b48744-ckjgl\" (UID: \"fe10d3b9-2dd4-4f73-8174-d888be2b8edb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.510623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510557 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006316eb-e088-4399-8d77-01501525c79c-tmp\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.510623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510587 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe10d3b9-2dd4-4f73-8174-d888be2b8edb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8475b48744-ckjgl\" (UID: \"fe10d3b9-2dd4-4f73-8174-d888be2b8edb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.510897 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510622 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8qqb\" (UniqueName: \"kubernetes.io/projected/69db2f4a-d983-40c1-950f-2d74d5ba4438-kube-api-access-z8qqb\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.510897 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510654 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.510897 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510688 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-hub\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.510897 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.510719 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.511603 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.511537 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/69db2f4a-d983-40c1-950f-2d74d5ba4438-additional-service-ca\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.512932 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.512883 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/006316eb-e088-4399-8d77-01501525c79c-tmp\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.523382 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.523360 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rc2p\" (UniqueName: \"kubernetes.io/projected/006316eb-e088-4399-8d77-01501525c79c-kube-api-access-2rc2p\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.529354 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.529333 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8qqb\" (UniqueName: \"kubernetes.io/projected/69db2f4a-d983-40c1-950f-2d74d5ba4438-kube-api-access-z8qqb\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.529915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.529870 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-hub\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.530443 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.530421 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.530918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.530899 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.531719 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.531680 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/69db2f4a-d983-40c1-950f-2d74d5ba4438-ca\") pod \"cluster-proxy-proxy-agent-665f47cdf7-gbkqm\" (UID: \"69db2f4a-d983-40c1-950f-2d74d5ba4438\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.531952 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.531933 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxmq6\" (UniqueName: \"kubernetes.io/projected/1582bf21-f4b1-4983-940a-76597189c439-kube-api-access-bxmq6\") pod \"downloads-7c97dc4555-8rpg2\" (UID: \"1582bf21-f4b1-4983-940a-76597189c439\") " pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:08.532040 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.531981 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/006316eb-e088-4399-8d77-01501525c79c-klusterlet-config\") pod \"klusterlet-addon-workmgr-78586b7d9c-tgxj8\" (UID: \"006316eb-e088-4399-8d77-01501525c79c\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.533580 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.533539 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jffv\" (UniqueName: \"kubernetes.io/projected/fe10d3b9-2dd4-4f73-8174-d888be2b8edb-kube-api-access-6jffv\") pod \"managed-serviceaccount-addon-agent-8475b48744-ckjgl\" (UID: \"fe10d3b9-2dd4-4f73-8174-d888be2b8edb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.534765 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.534746 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe10d3b9-2dd4-4f73-8174-d888be2b8edb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-8475b48744-ckjgl\" (UID: \"fe10d3b9-2dd4-4f73-8174-d888be2b8edb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.581186 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.581031 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" Jul 24 23:30:08.597997 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.597975 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:08.619214 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.618714 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:08.632033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.632012 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" Jul 24 23:30:08.920046 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:08.919952 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-8rpg2"] Jul 24 23:30:09.051947 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:09.051909 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1582bf21_f4b1_4983_940a_76597189c439.slice/crio-85f8af09aa25932151536621873c184dc921bf83138d87ab77e364cdd833de38 WatchSource:0}: Error finding container 85f8af09aa25932151536621873c184dc921bf83138d87ab77e364cdd833de38: Status 404 returned error can't find the container with id 85f8af09aa25932151536621873c184dc921bf83138d87ab77e364cdd833de38 Jul 24 23:30:09.163979 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.163885 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm"] Jul 24 23:30:09.168631 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.168285 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl"] Jul 24 23:30:09.169267 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.169226 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8"] Jul 24 23:30:09.342668 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.342587 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4s7v8" event={"ID":"5ff1b64c-69a3-4c3d-846a-3494d996bed5","Type":"ContainerStarted","Data":"aa2f78504be223024a9df39b94ece5e4a3feca2855d67d22a4ee8b7621513c66"} Jul 24 23:30:09.347256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.346358 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-8rpg2" event={"ID":"1582bf21-f4b1-4983-940a-76597189c439","Type":"ContainerStarted","Data":"85f8af09aa25932151536621873c184dc921bf83138d87ab77e364cdd833de38"} Jul 24 23:30:09.347256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.347106 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:09.348717 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:09.348495 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-59f648499-fvcgf" Jul 24 23:30:10.992077 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:10.992041 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe10d3b9_2dd4_4f73_8174_d888be2b8edb.slice/crio-361a6eb8b4fe37b3a40c379e528df883998b2bb6a0a68bf4b1b250ac5d0c2184 WatchSource:0}: Error finding container 361a6eb8b4fe37b3a40c379e528df883998b2bb6a0a68bf4b1b250ac5d0c2184: Status 404 returned error can't find the container with id 361a6eb8b4fe37b3a40c379e528df883998b2bb6a0a68bf4b1b250ac5d0c2184 Jul 24 23:30:11.354867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:11.354834 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" event={"ID":"fe10d3b9-2dd4-4f73-8174-d888be2b8edb","Type":"ContainerStarted","Data":"361a6eb8b4fe37b3a40c379e528df883998b2bb6a0a68bf4b1b250ac5d0c2184"} Jul 24 23:30:12.244960 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:12.244365 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69db2f4a_d983_40c1_950f_2d74d5ba4438.slice/crio-ede3052f7265aa6e64ecc0b1a80a4bdbaa79778395d9ff78453858e838140277 WatchSource:0}: Error finding container ede3052f7265aa6e64ecc0b1a80a4bdbaa79778395d9ff78453858e838140277: Status 404 returned error can't find the container with id ede3052f7265aa6e64ecc0b1a80a4bdbaa79778395d9ff78453858e838140277 Jul 24 23:30:12.244960 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:12.244679 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod006316eb_e088_4399_8d77_01501525c79c.slice/crio-266bb97fed3cc39546729e387bbcda7d2f46d04a065c7ecdd2dff9d43fdc1ac0 WatchSource:0}: Error finding container 266bb97fed3cc39546729e387bbcda7d2f46d04a065c7ecdd2dff9d43fdc1ac0: Status 404 returned error can't find the container with id 266bb97fed3cc39546729e387bbcda7d2f46d04a065c7ecdd2dff9d43fdc1ac0 Jul 24 23:30:12.361307 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:12.361272 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" event={"ID":"006316eb-e088-4399-8d77-01501525c79c","Type":"ContainerStarted","Data":"266bb97fed3cc39546729e387bbcda7d2f46d04a065c7ecdd2dff9d43fdc1ac0"} Jul 24 23:30:12.362568 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:12.362535 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" event={"ID":"69db2f4a-d983-40c1-950f-2d74d5ba4438","Type":"ContainerStarted","Data":"ede3052f7265aa6e64ecc0b1a80a4bdbaa79778395d9ff78453858e838140277"} Jul 24 23:30:17.401518 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.401326 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" event={"ID":"477c6c0e-fc47-4c72-adbe-f697c23204ac","Type":"ContainerStarted","Data":"5c30f4c5426d21e13434507c0c48b6a9e72145a4bb8e1c309e4d688486d97b76"} Jul 24 23:30:17.409967 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.406375 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m8fd" event={"ID":"daba6781-d561-4345-a105-3f0f93979fe5","Type":"ContainerStarted","Data":"aa0afb9a52d44053e09803176b83bbe185763e360bc86717fc095a5cc9c166c4"} Jul 24 23:30:17.423824 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.423767 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-h9j6b" event={"ID":"5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52","Type":"ContainerStarted","Data":"d9b9f99cca2690934e700976fd343bb9c27dc3ba1da8f9dfe6f808d0bd114ec9"} Jul 24 23:30:17.426790 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.426741 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-v4jld" podStartSLOduration=45.698652922 podStartE2EDuration="54.426725251s" totalImagesPullingTime="8.728072329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:17.42482625 +0000 UTC m=+77.117667083" watchObservedRunningTime="2026-07-24 23:30:17.426725251 +0000 UTC m=+77.119566081" Jul 24 23:30:17.434224 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.433679 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" event={"ID":"cbc6bcda-8035-4b85-b977-4a4567482781","Type":"ContainerStarted","Data":"fcd19bbb4e81f595cbf720a3359fbd5beaeb7155454a101ef4dc592d8346a84f"} Jul 24 23:30:17.434224 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.433727 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" event={"ID":"cbc6bcda-8035-4b85-b977-4a4567482781","Type":"ContainerStarted","Data":"25d5753d3fe4c9ba507eaea899158948ef73a563f615f5f05d66bf02476fd04f"} Jul 24 23:30:17.443264 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.442882 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4s7v8" event={"ID":"5ff1b64c-69a3-4c3d-846a-3494d996bed5","Type":"ContainerStarted","Data":"03a1bcb5e0408020c8bab33cb73e8faeb19c253cd195c24b981de2fd37e040c5"} Jul 24 23:30:17.448093 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.448054 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p54z5" event={"ID":"32fd7b44-4b9c-4733-a471-3316a037d9f9","Type":"ContainerStarted","Data":"ba3be10d660b0e2b0e7e79c27612c68a0b6cf161790f0e728d0490743428dfba"} Jul 24 23:30:17.448186 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.448100 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p54z5" event={"ID":"32fd7b44-4b9c-4733-a471-3316a037d9f9","Type":"ContainerStarted","Data":"feebed4d0f086572180cae72cf8a1058097790d3893d368882f1917f63bea7b1"} Jul 24 23:30:17.448603 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.448582 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p54z5" Jul 24 23:30:17.456850 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.456793 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-74wqg" podStartSLOduration=46.730195526 podStartE2EDuration="55.456776759s" totalImagesPullingTime="8.726581233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:17.45422406 +0000 UTC m=+77.147064889" watchObservedRunningTime="2026-07-24 23:30:17.456776759 +0000 UTC m=+77.149617588" Jul 24 23:30:17.458764 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.457333 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-h9j6b" podStartSLOduration=67.216306095 podStartE2EDuration="1m16.457325163s" totalImagesPullingTime="9.241019068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:17.440396001 +0000 UTC m=+77.133236832" watchObservedRunningTime="2026-07-24 23:30:17.457325163 +0000 UTC m=+77.150165972" Jul 24 23:30:17.471641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.471596 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4s7v8" podStartSLOduration=74.34993501 podStartE2EDuration="1m16.471579927s" totalImagesPullingTime="2.121644917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:17.470800101 +0000 UTC m=+77.163640943" watchObservedRunningTime="2026-07-24 23:30:17.471579927 +0000 UTC m=+77.164420752" Jul 24 23:30:17.498265 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:17.493469 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p54z5" podStartSLOduration=33.697852414 podStartE2EDuration="42.493452617s" totalImagesPullingTime="8.795600203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:17.488571044 +0000 UTC m=+77.181411885" watchObservedRunningTime="2026-07-24 23:30:17.493452617 +0000 UTC m=+77.186293446" Jul 24 23:30:27.310787 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.310729 2586 patch_prober.go:28] interesting pod/image-registry-64b5b8f786-lj78n container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 23:30:27.311226 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.310827 2586 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" podUID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 23:30:27.595475 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.594809 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mh72p"] Jul 24 23:30:27.659266 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.658438 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.663730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.663165 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 23:30:27.663730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.663455 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 23:30:27.663730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.663475 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 23:30:27.663966 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.663873 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n4stc\"" Jul 24 23:30:27.664095 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.664079 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.702946 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-textfile\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703007 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccbpj\" (UniqueName: \"kubernetes.io/projected/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-kube-api-access-ccbpj\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703054 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703084 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-sys\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703126 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-wtmp\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703159 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-tls\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703181 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-metrics-client-ca\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703217 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-root\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.706641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.703277 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804506 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804465 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-textfile\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804712 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804534 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccbpj\" (UniqueName: \"kubernetes.io/projected/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-kube-api-access-ccbpj\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804712 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804588 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804712 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804613 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-sys\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804712 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804658 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-wtmp\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804712 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804690 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-tls\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804874 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804713 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-metrics-client-ca\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804874 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804750 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-root\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.804874 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.804781 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.806107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.805545 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-accelerators-collector-config\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.806107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.805757 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-textfile\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.806107 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:30:27.805859 2586 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 23:30:27.806107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.805872 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-sys\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.806107 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:30:27.805913 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-tls podName:a422bd0e-5604-4ff7-93c0-01b945fe2a9c nodeName:}" failed. No retries permitted until 2026-07-24 23:30:28.305894606 +0000 UTC m=+87.998735424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-tls") pod "node-exporter-mh72p" (UID: "a422bd0e-5604-4ff7-93c0-01b945fe2a9c") : secret "node-exporter-tls" not found Jul 24 23:30:27.806107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.805994 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-wtmp\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.806560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.806139 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-root\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.806560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.806465 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-metrics-client-ca\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.809043 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.809000 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:27.814767 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:27.814745 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccbpj\" (UniqueName: \"kubernetes.io/projected/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-kube-api-access-ccbpj\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:28.310690 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:28.310651 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-tls\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:28.313577 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:28.313545 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a422bd0e-5604-4ff7-93c0-01b945fe2a9c-node-exporter-tls\") pod \"node-exporter-mh72p\" (UID: \"a422bd0e-5604-4ff7-93c0-01b945fe2a9c\") " pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:28.466298 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:28.466261 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p54z5" Jul 24 23:30:28.576736 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:28.576647 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mh72p" Jul 24 23:30:29.352075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:29.352046 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:31.553065 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:31.553031 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b5b8f786-lj78n"] Jul 24 23:30:33.024365 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:30:33.024339 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda422bd0e_5604_4ff7_93c0_01b945fe2a9c.slice/crio-de90731715794ffb2991c7d737efc773116f3043b9bcea6e8d4e22171dc68a4b WatchSource:0}: Error finding container de90731715794ffb2991c7d737efc773116f3043b9bcea6e8d4e22171dc68a4b: Status 404 returned error can't find the container with id de90731715794ffb2991c7d737efc773116f3043b9bcea6e8d4e22171dc68a4b Jul 24 23:30:33.540145 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.540108 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh72p" event={"ID":"a422bd0e-5604-4ff7-93c0-01b945fe2a9c","Type":"ContainerStarted","Data":"de90731715794ffb2991c7d737efc773116f3043b9bcea6e8d4e22171dc68a4b"} Jul 24 23:30:33.542331 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.542278 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" event={"ID":"fe10d3b9-2dd4-4f73-8174-d888be2b8edb","Type":"ContainerStarted","Data":"99d799d038b533b16ef59f667510cbea21b0da2d9136d6bc332cbcd9b8f7c5ad"} Jul 24 23:30:33.545404 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.545342 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8m8fd" event={"ID":"daba6781-d561-4345-a105-3f0f93979fe5","Type":"ContainerStarted","Data":"4ca1fd37670be58c0fceee191483e7ae7071f5c3d14802e3cbfed01ab1fae678"} Jul 24 23:30:33.547463 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.547440 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" event={"ID":"006316eb-e088-4399-8d77-01501525c79c","Type":"ContainerStarted","Data":"1c9b090611addd1b4286d26d98c367b550d0065d276d064d75921b393e355653"} Jul 24 23:30:33.547614 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.547593 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:33.549928 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.549908 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" Jul 24 23:30:33.551059 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.550914 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" event={"ID":"69db2f4a-d983-40c1-950f-2d74d5ba4438","Type":"ContainerStarted","Data":"c900331d3ca84df727af56247ce8abb0cdd52ce3577fd6b9f469ffe19ed3820d"} Jul 24 23:30:33.551059 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.550949 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" event={"ID":"69db2f4a-d983-40c1-950f-2d74d5ba4438","Type":"ContainerStarted","Data":"babca42a93a5594b076bfc3662ff6acc7cc0392f0cb3759c7428e976fa95e9f3"} Jul 24 23:30:33.551059 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.550967 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" event={"ID":"69db2f4a-d983-40c1-950f-2d74d5ba4438","Type":"ContainerStarted","Data":"10b1f199edb4c4b571e13e8b72e64bce67909382f60d10630e6541c801af2ed5"} Jul 24 23:30:33.553632 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.553593 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-8rpg2" event={"ID":"1582bf21-f4b1-4983-940a-76597189c439","Type":"ContainerStarted","Data":"4e38cccdc13e808be6751f7a25ea99e70416f22593f2efe4f5426e6c73203a45"} Jul 24 23:30:33.553814 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.553794 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:33.554861 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.554828 2586 patch_prober.go:28] interesting pod/downloads-7c97dc4555-8rpg2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.23:8080/\": dial tcp 10.132.0.23:8080: connect: connection refused" start-of-body= Jul 24 23:30:33.554948 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.554879 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-7c97dc4555-8rpg2" podUID="1582bf21-f4b1-4983-940a-76597189c439" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.23:8080/\": dial tcp 10.132.0.23:8080: connect: connection refused" Jul 24 23:30:33.558278 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.557769 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-8475b48744-ckjgl" podStartSLOduration=3.536786694 podStartE2EDuration="25.557758838s" totalImagesPullingTime="22.020972144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:33.557317611 +0000 UTC m=+93.250158444" watchObservedRunningTime="2026-07-24 23:30:33.557758838 +0000 UTC m=+93.250599666" Jul 24 23:30:33.573906 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.573844 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-78586b7d9c-tgxj8" podStartSLOduration=4.800797716 podStartE2EDuration="25.573824614s" totalImagesPullingTime="20.773026898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:33.573197785 +0000 UTC m=+93.266038613" watchObservedRunningTime="2026-07-24 23:30:33.573824614 +0000 UTC m=+93.266665446" Jul 24 23:30:33.590866 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.590811 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-665f47cdf7-gbkqm" podStartSLOduration=4.805228923 podStartE2EDuration="25.5907955s" totalImagesPullingTime="20.785566577s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:33.590324281 +0000 UTC m=+93.283165111" watchObservedRunningTime="2026-07-24 23:30:33.5907955 +0000 UTC m=+93.283636328" Jul 24 23:30:33.607130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:33.606993 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8m8fd" podStartSLOduration=17.177015073 podStartE2EDuration="48.606972209s" totalImagesPullingTime="31.429957136s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 23:29:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:33.605751109 +0000 UTC m=+93.298591938" watchObservedRunningTime="2026-07-24 23:30:33.606972209 +0000 UTC m=+93.299813038" Jul 24 23:30:34.575214 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:34.575183 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-8rpg2" Jul 24 23:30:34.597293 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:34.597223 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-8rpg2" podStartSLOduration=2.359246407 podStartE2EDuration="26.597201825s" totalImagesPullingTime="24.237955418s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:30:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:33.628099738 +0000 UTC m=+93.320940564" watchObservedRunningTime="2026-07-24 23:30:34.597201825 +0000 UTC m=+94.290042654" Jul 24 23:30:35.566187 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:35.566154 2586 generic.go:356] "Generic (PLEG): container finished" podID="a422bd0e-5604-4ff7-93c0-01b945fe2a9c" containerID="18b62ec51d55e87edec68dd2399d1608aa65014f47ea773f1c55b98ddb346bc5" exitCode=0 Jul 24 23:30:35.566401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:35.566265 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh72p" event={"ID":"a422bd0e-5604-4ff7-93c0-01b945fe2a9c","Type":"ContainerDied","Data":"18b62ec51d55e87edec68dd2399d1608aa65014f47ea773f1c55b98ddb346bc5"} Jul 24 23:30:36.574899 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:36.574855 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh72p" event={"ID":"a422bd0e-5604-4ff7-93c0-01b945fe2a9c","Type":"ContainerStarted","Data":"6da7049c13be740fd28135df7b5661f765886f9cebff18e89dade143df349ff3"} Jul 24 23:30:36.575391 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:36.574964 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mh72p" event={"ID":"a422bd0e-5604-4ff7-93c0-01b945fe2a9c","Type":"ContainerStarted","Data":"5ecf10d88f7069eb57fba44087cdfc425495e86f68b84285713466f816cc3988"} Jul 24 23:30:36.593755 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:36.593701 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mh72p" podStartSLOduration=7.903395112 podStartE2EDuration="9.593685986s" totalImagesPullingTime="1.601573152s" totalInitContainerRuntime="88.717722ms" isStatefulPod=true podCreationTimestamp="2026-07-24 23:30:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:30:36.592212043 +0000 UTC m=+96.285052872" watchObservedRunningTime="2026-07-24 23:30:36.593685986 +0000 UTC m=+96.286526813" Jul 24 23:30:38.336430 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:38.336389 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lhlft" Jul 24 23:30:50.637518 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:50.637481 2586 generic.go:356] "Generic (PLEG): container finished" podID="c76723fb-9238-4421-bc73-5bf67c9604d6" containerID="9a0dcaf96816599649d140d6f8cbb15c39057c09c9bef1cde7da30fa06fb30f5" exitCode=0 Jul 24 23:30:50.637977 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:50.637556 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" event={"ID":"c76723fb-9238-4421-bc73-5bf67c9604d6","Type":"ContainerDied","Data":"9a0dcaf96816599649d140d6f8cbb15c39057c09c9bef1cde7da30fa06fb30f5"} Jul 24 23:30:50.637977 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:50.637864 2586 scope.go:117] "RemoveContainer" containerID="9a0dcaf96816599649d140d6f8cbb15c39057c09c9bef1cde7da30fa06fb30f5" Jul 24 23:30:51.643040 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:51.642994 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-8lhwc" event={"ID":"c76723fb-9238-4421-bc73-5bf67c9604d6","Type":"ContainerStarted","Data":"91e61a59669d332c63485778d6fa3c56548f5ccf30e174309580bf6351c631d6"} Jul 24 23:30:56.579891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.579857 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" podUID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" containerName="registry" containerID="cri-o://90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72" gracePeriod=30 Jul 24 23:30:56.662816 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.662789 2586 generic.go:356] "Generic (PLEG): container finished" podID="e8b72a9a-44f2-4d63-866b-62cccd430125" containerID="021fabd147e1cef1e3ee0d0548c817b34ab8201dc854e82e8162d70dd38eb635" exitCode=0 Jul 24 23:30:56.662916 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.662876 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" event={"ID":"e8b72a9a-44f2-4d63-866b-62cccd430125","Type":"ContainerDied","Data":"021fabd147e1cef1e3ee0d0548c817b34ab8201dc854e82e8162d70dd38eb635"} Jul 24 23:30:56.663174 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.663159 2586 scope.go:117] "RemoveContainer" containerID="021fabd147e1cef1e3ee0d0548c817b34ab8201dc854e82e8162d70dd38eb635" Jul 24 23:30:56.870644 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.870614 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:56.965866 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.965841 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-installation-pull-secrets\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.965893 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.965910 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-bound-sa-token\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.965934 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/957f8e9f-e8fd-4113-9f7c-058c8b1755be-ca-trust-extracted\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.965956 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-certificates\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.965974 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-image-registry-private-configuration\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966278 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.966023 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj64f\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-kube-api-access-rj64f\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966278 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.966058 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-trusted-ca\") pod \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\" (UID: \"957f8e9f-e8fd-4113-9f7c-058c8b1755be\") " Jul 24 23:30:56.966625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.966576 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:30:56.966625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.966607 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:30:56.968666 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.968629 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:30:56.968956 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.968935 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:30:56.969106 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.969075 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-kube-api-access-rj64f" (OuterVolumeSpecName: "kube-api-access-rj64f") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "kube-api-access-rj64f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:56.969183 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.969157 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:56.969941 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.969922 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:30:56.976738 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:56.976712 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/957f8e9f-e8fd-4113-9f7c-058c8b1755be-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "957f8e9f-e8fd-4113-9f7c-058c8b1755be" (UID: "957f8e9f-e8fd-4113-9f7c-058c8b1755be"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:30:57.066971 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.066950 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj64f\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-kube-api-access-rj64f\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.066971 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.066973 2586 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-trusted-ca\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.067094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.066983 2586 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-installation-pull-secrets\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.067094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.066992 2586 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-tls\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.067094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.067000 2586 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/957f8e9f-e8fd-4113-9f7c-058c8b1755be-bound-sa-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.067094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.067007 2586 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/957f8e9f-e8fd-4113-9f7c-058c8b1755be-ca-trust-extracted\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.067094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.067014 2586 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/957f8e9f-e8fd-4113-9f7c-058c8b1755be-registry-certificates\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.067094 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.067024 2586 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/957f8e9f-e8fd-4113-9f7c-058c8b1755be-image-registry-private-configuration\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:30:57.671723 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.671681 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88" event={"ID":"e8b72a9a-44f2-4d63-866b-62cccd430125","Type":"ContainerStarted","Data":"6ddfe1af800d4012039ce24e17187ab33488e7e2aa4b97ff0994d9d9a28f5639"} Jul 24 23:30:57.673009 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.672983 2586 generic.go:356] "Generic (PLEG): container finished" podID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" containerID="90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72" exitCode=0 Jul 24 23:30:57.673130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.673022 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" Jul 24 23:30:57.673130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.673064 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" event={"ID":"957f8e9f-e8fd-4113-9f7c-058c8b1755be","Type":"ContainerDied","Data":"90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72"} Jul 24 23:30:57.673130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.673114 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64b5b8f786-lj78n" event={"ID":"957f8e9f-e8fd-4113-9f7c-058c8b1755be","Type":"ContainerDied","Data":"082069b1f8cf419c640a69e8f11a49b82a3cb1dcb305b23c05399b789e186dd6"} Jul 24 23:30:57.673261 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.673137 2586 scope.go:117] "RemoveContainer" containerID="90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72" Jul 24 23:30:57.681750 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.681725 2586 scope.go:117] "RemoveContainer" containerID="90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72" Jul 24 23:30:57.682015 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:30:57.681990 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72\": container with ID starting with 90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72 not found: ID does not exist" containerID="90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72" Jul 24 23:30:57.682064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.682026 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72"} err="failed to get container status \"90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72\": rpc error: code = NotFound desc = could not find container \"90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72\": container with ID starting with 90cee7391fcdb4bba5b21de3b3a1db008d2ac7c5d4e438cf6623a713a0e78a72 not found: ID does not exist" Jul 24 23:30:57.701448 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.701424 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64b5b8f786-lj78n"] Jul 24 23:30:57.707728 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:57.707706 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64b5b8f786-lj78n"] Jul 24 23:30:58.785531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:30:58.785499 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" path="/var/lib/kubelet/pods/957f8e9f-e8fd-4113-9f7c-058c8b1755be/volumes" Jul 24 23:31:05.707419 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:31:05.707380 2586 generic.go:356] "Generic (PLEG): container finished" podID="2d924791-ad88-44a5-8a8d-90fce288b00a" containerID="35fc32e419a23465132d14778d0edc8cb62d7641d787ad37a712db413f5f2739" exitCode=0 Jul 24 23:31:05.707834 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:31:05.707445 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" event={"ID":"2d924791-ad88-44a5-8a8d-90fce288b00a","Type":"ContainerDied","Data":"35fc32e419a23465132d14778d0edc8cb62d7641d787ad37a712db413f5f2739"} Jul 24 23:31:05.707834 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:31:05.707762 2586 scope.go:117] "RemoveContainer" containerID="35fc32e419a23465132d14778d0edc8cb62d7641d787ad37a712db413f5f2739" Jul 24 23:31:06.712421 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:31:06.712387 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9xj5h" event={"ID":"2d924791-ad88-44a5-8a8d-90fce288b00a","Type":"ContainerStarted","Data":"136670c31df4759b2389f7da8533788cd84da554c5db46c2a000ce376aef4e82"} Jul 24 23:33:52.740935 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.740895 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r"] Jul 24 23:33:52.741407 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.741211 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" containerName="registry" Jul 24 23:33:52.741407 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.741221 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" containerName="registry" Jul 24 23:33:52.741407 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.741308 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="957f8e9f-e8fd-4113-9f7c-058c8b1755be" containerName="registry" Jul 24 23:33:52.743099 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.743082 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.745645 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.745623 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 23:33:52.745886 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.745866 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 23:33:52.745942 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.745924 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 23:33:52.746130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.746117 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 23:33:52.746471 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.746453 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-r9vz8\"" Jul 24 23:33:52.763563 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.763535 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r"] Jul 24 23:33:52.895488 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.895450 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3576e52-c673-44cf-a992-46b94db07d5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.895629 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.895503 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3576e52-c673-44cf-a992-46b94db07d5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.895629 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.895556 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3576e52-c673-44cf-a992-46b94db07d5a-manifests\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.895629 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.895577 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2pms\" (UniqueName: \"kubernetes.io/projected/a3576e52-c673-44cf-a992-46b94db07d5a-kube-api-access-d2pms\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.895734 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.895645 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3576e52-c673-44cf-a992-46b94db07d5a-tmp\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.995985 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.995910 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3576e52-c673-44cf-a992-46b94db07d5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.995985 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.995957 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3576e52-c673-44cf-a992-46b94db07d5a-manifests\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.995985 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.995975 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2pms\" (UniqueName: \"kubernetes.io/projected/a3576e52-c673-44cf-a992-46b94db07d5a-kube-api-access-d2pms\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.996228 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.996000 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3576e52-c673-44cf-a992-46b94db07d5a-tmp\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.996228 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.996037 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3576e52-c673-44cf-a992-46b94db07d5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.996400 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.996375 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a3576e52-c673-44cf-a992-46b94db07d5a-manifests\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.998271 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.998226 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a3576e52-c673-44cf-a992-46b94db07d5a-tmp\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.998474 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.998458 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a3576e52-c673-44cf-a992-46b94db07d5a-webhook-cert\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:52.998513 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:52.998469 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a3576e52-c673-44cf-a992-46b94db07d5a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:53.007495 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:53.007469 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2pms\" (UniqueName: \"kubernetes.io/projected/a3576e52-c673-44cf-a992-46b94db07d5a-kube-api-access-d2pms\") pod \"opendatahub-operator-controller-manager-69778f9898-vxr7r\" (UID: \"a3576e52-c673-44cf-a992-46b94db07d5a\") " pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:53.057409 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:53.057384 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:53.210814 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:53.210788 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r"] Jul 24 23:33:53.213374 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:33:53.213352 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3576e52_c673_44cf_a992_46b94db07d5a.slice/crio-6ec045330ad49e0f6f43b6054e08f45bab525133e2801117e66ed40901b8309e WatchSource:0}: Error finding container 6ec045330ad49e0f6f43b6054e08f45bab525133e2801117e66ed40901b8309e: Status 404 returned error can't find the container with id 6ec045330ad49e0f6f43b6054e08f45bab525133e2801117e66ed40901b8309e Jul 24 23:33:53.344278 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:53.344215 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" event={"ID":"a3576e52-c673-44cf-a992-46b94db07d5a","Type":"ContainerStarted","Data":"6ec045330ad49e0f6f43b6054e08f45bab525133e2801117e66ed40901b8309e"} Jul 24 23:33:56.360064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:56.360037 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" event={"ID":"a3576e52-c673-44cf-a992-46b94db07d5a","Type":"ContainerStarted","Data":"60d63f7bd053982d9fa1cced43ce2f0f580d7a6dca921ddba76da6117e1eab23"} Jul 24 23:33:57.365595 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:57.365564 2586 generic.go:356] "Generic (PLEG): container finished" podID="a3576e52-c673-44cf-a992-46b94db07d5a" containerID="60d63f7bd053982d9fa1cced43ce2f0f580d7a6dca921ddba76da6117e1eab23" exitCode=0 Jul 24 23:33:57.365939 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:57.365618 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" event={"ID":"a3576e52-c673-44cf-a992-46b94db07d5a","Type":"ContainerDied","Data":"60d63f7bd053982d9fa1cced43ce2f0f580d7a6dca921ddba76da6117e1eab23"} Jul 24 23:33:58.373165 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:58.373123 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" event={"ID":"a3576e52-c673-44cf-a992-46b94db07d5a","Type":"ContainerStarted","Data":"092e2d91483e015c1d58298b24f97bd1319fedce69e2dd3faae99b0cac3578c2"} Jul 24 23:33:58.373649 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:58.373280 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:33:58.395876 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:33:58.395831 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" podStartSLOduration=2.047776837 podStartE2EDuration="6.395818903s" totalImagesPullingTime="3.308786639s" totalInitContainerRuntime="1.039255427s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:33:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:33:58.394392037 +0000 UTC m=+298.087232865" watchObservedRunningTime="2026-07-24 23:33:58.395818903 +0000 UTC m=+298.088659731" Jul 24 23:34:00.721099 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:00.721069 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:34:00.728962 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:00.728931 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:34:00.734863 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:00.734844 2586 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 23:34:09.380159 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:09.380122 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-69778f9898-vxr7r" Jul 24 23:34:12.389393 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.389353 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dx2nm"] Jul 24 23:34:12.392176 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.392156 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:12.394490 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.394466 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 24 23:34:12.394592 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.394497 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-79szn\"" Jul 24 23:34:12.400449 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.400422 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dx2nm"] Jul 24 23:34:12.446864 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.446834 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02cba9-3c04-4368-8307-27ee04bff695-tls-certs\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:12.447015 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.446899 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nhc\" (UniqueName: \"kubernetes.io/projected/de02cba9-3c04-4368-8307-27ee04bff695-kube-api-access-j9nhc\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:12.547317 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.547282 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nhc\" (UniqueName: \"kubernetes.io/projected/de02cba9-3c04-4368-8307-27ee04bff695-kube-api-access-j9nhc\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:12.547489 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.547381 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02cba9-3c04-4368-8307-27ee04bff695-tls-certs\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:12.547556 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:12.547507 2586 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 24 23:34:12.547608 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:12.547587 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de02cba9-3c04-4368-8307-27ee04bff695-tls-certs podName:de02cba9-3c04-4368-8307-27ee04bff695 nodeName:}" failed. No retries permitted until 2026-07-24 23:34:13.047564891 +0000 UTC m=+312.740405721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/de02cba9-3c04-4368-8307-27ee04bff695-tls-certs") pod "model-serving-api-54d89bd5fc-dx2nm" (UID: "de02cba9-3c04-4368-8307-27ee04bff695") : secret "model-serving-api-tls" not found Jul 24 23:34:12.557722 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:12.557690 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nhc\" (UniqueName: \"kubernetes.io/projected/de02cba9-3c04-4368-8307-27ee04bff695-kube-api-access-j9nhc\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:13.051259 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:13.051197 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02cba9-3c04-4368-8307-27ee04bff695-tls-certs\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:13.053640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:13.053614 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/de02cba9-3c04-4368-8307-27ee04bff695-tls-certs\") pod \"model-serving-api-54d89bd5fc-dx2nm\" (UID: \"de02cba9-3c04-4368-8307-27ee04bff695\") " pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:13.303810 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:13.303731 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:13.436658 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:13.436629 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dx2nm"] Jul 24 23:34:13.441417 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:34:13.441387 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde02cba9_3c04_4368_8307_27ee04bff695.slice/crio-2abf15a393794480e62be2c493ad18ff08ccfc3e603eeff12d425e8e9cbc37cf WatchSource:0}: Error finding container 2abf15a393794480e62be2c493ad18ff08ccfc3e603eeff12d425e8e9cbc37cf: Status 404 returned error can't find the container with id 2abf15a393794480e62be2c493ad18ff08ccfc3e603eeff12d425e8e9cbc37cf Jul 24 23:34:13.443314 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:13.443294 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:34:14.443290 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.443255 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" event={"ID":"de02cba9-3c04-4368-8307-27ee04bff695","Type":"ContainerStarted","Data":"2abf15a393794480e62be2c493ad18ff08ccfc3e603eeff12d425e8e9cbc37cf"} Jul 24 23:34:14.569484 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.569453 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv"] Jul 24 23:34:14.572180 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.572150 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.575323 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.575301 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-krzl2\"" Jul 24 23:34:14.575749 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.575647 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 24 23:34:14.576063 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.576042 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 24 23:34:14.593150 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.593114 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv"] Jul 24 23:34:14.663416 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.663379 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662651c3-c530-477e-9071-4aa47747f54d-tls-certs\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.663416 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.663419 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/662651c3-c530-477e-9071-4aa47747f54d-tmp\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.663617 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.663444 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzfbw\" (UniqueName: \"kubernetes.io/projected/662651c3-c530-477e-9071-4aa47747f54d-kube-api-access-vzfbw\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.764741 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.764645 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662651c3-c530-477e-9071-4aa47747f54d-tls-certs\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.764741 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.764699 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/662651c3-c530-477e-9071-4aa47747f54d-tmp\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.764741 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.764726 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzfbw\" (UniqueName: \"kubernetes.io/projected/662651c3-c530-477e-9071-4aa47747f54d-kube-api-access-vzfbw\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.765022 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:14.764807 2586 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 24 23:34:14.765022 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:14.764904 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/662651c3-c530-477e-9071-4aa47747f54d-tls-certs podName:662651c3-c530-477e-9071-4aa47747f54d nodeName:}" failed. No retries permitted until 2026-07-24 23:34:15.264881603 +0000 UTC m=+314.957722414 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/662651c3-c530-477e-9071-4aa47747f54d-tls-certs") pod "kube-auth-proxy-6d7d7d465-f2xmv" (UID: "662651c3-c530-477e-9071-4aa47747f54d") : secret "kube-auth-proxy-tls" not found Jul 24 23:34:14.767953 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.767900 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/662651c3-c530-477e-9071-4aa47747f54d-tmp\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:14.774264 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:14.774228 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzfbw\" (UniqueName: \"kubernetes.io/projected/662651c3-c530-477e-9071-4aa47747f54d-kube-api-access-vzfbw\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:15.268053 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.268028 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662651c3-c530-477e-9071-4aa47747f54d-tls-certs\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:15.270473 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.270450 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/662651c3-c530-477e-9071-4aa47747f54d-tls-certs\") pod \"kube-auth-proxy-6d7d7d465-f2xmv\" (UID: \"662651c3-c530-477e-9071-4aa47747f54d\") " pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:15.449853 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.449817 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" event={"ID":"de02cba9-3c04-4368-8307-27ee04bff695","Type":"ContainerStarted","Data":"4689a6b1cdb9e15d46a6784591e3744cd09e71c716809005c49a591d2426dd8e"} Jul 24 23:34:15.450324 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.449890 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:15.471907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.471846 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" podStartSLOduration=1.663368489 podStartE2EDuration="3.471830741s" totalImagesPullingTime="1.808462252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:15.469082025 +0000 UTC m=+315.161922853" watchObservedRunningTime="2026-07-24 23:34:15.471830741 +0000 UTC m=+315.164671569" Jul 24 23:34:15.485499 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.485472 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" Jul 24 23:34:15.628499 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.628463 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv"] Jul 24 23:34:15.631396 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:34:15.631366 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod662651c3_c530_477e_9071_4aa47747f54d.slice/crio-7f2d297ae80956d6fff398b44a06e2cc201b654f28e558021101013ad6868175 WatchSource:0}: Error finding container 7f2d297ae80956d6fff398b44a06e2cc201b654f28e558021101013ad6868175: Status 404 returned error can't find the container with id 7f2d297ae80956d6fff398b44a06e2cc201b654f28e558021101013ad6868175 Jul 24 23:34:15.687355 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.684878 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9"] Jul 24 23:34:15.692397 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.692368 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.694563 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.694538 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 24 23:34:15.694719 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.694574 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 24 23:34:15.694719 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.694582 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 23:34:15.694719 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.694595 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 24 23:34:15.694882 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.694863 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hpn6q\"" Jul 24 23:34:15.702494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.702471 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9"] Jul 24 23:34:15.770901 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.770870 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4517be-0e39-481b-9428-0fcad57eb605-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.770901 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.770905 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjj4v\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-kube-api-access-jjj4v\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.771108 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.770925 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.771108 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.770983 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.771108 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.771016 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.771108 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.771040 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.771108 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.771059 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872030 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872002 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872045 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872164 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4517be-0e39-481b-9428-0fcad57eb605-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872356 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872192 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjj4v\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-kube-api-access-jjj4v\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872356 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872221 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872356 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872277 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872356 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872306 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.872938 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.872812 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.874684 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.874656 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4517be-0e39-481b-9428-0fcad57eb605-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.874786 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.874726 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.875036 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.875019 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.875098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.875041 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.884209 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.884186 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:15.884651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:15.884631 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjj4v\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-kube-api-access-jjj4v\") pod \"istiod-openshift-gateway-7b7995d5cd-6zjr9\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:16.002515 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:16.002472 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:16.143069 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:16.143040 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9"] Jul 24 23:34:16.144893 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:34:16.144860 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f4517be_0e39_481b_9428_0fcad57eb605.slice/crio-0eee0f43dc68bb82e731218a3e76d28749bb4ac65873b7f77ad67ff589c36686 WatchSource:0}: Error finding container 0eee0f43dc68bb82e731218a3e76d28749bb4ac65873b7f77ad67ff589c36686: Status 404 returned error can't find the container with id 0eee0f43dc68bb82e731218a3e76d28749bb4ac65873b7f77ad67ff589c36686 Jul 24 23:34:16.459855 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:16.459765 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" event={"ID":"662651c3-c530-477e-9071-4aa47747f54d","Type":"ContainerStarted","Data":"7f2d297ae80956d6fff398b44a06e2cc201b654f28e558021101013ad6868175"} Jul 24 23:34:16.461281 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:16.461226 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" event={"ID":"9f4517be-0e39-481b-9428-0fcad57eb605","Type":"ContainerStarted","Data":"0eee0f43dc68bb82e731218a3e76d28749bb4ac65873b7f77ad67ff589c36686"} Jul 24 23:34:19.205968 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.205930 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:34:19.206368 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.206031 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:34:19.483308 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.483207 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" event={"ID":"662651c3-c530-477e-9071-4aa47747f54d","Type":"ContainerStarted","Data":"8bf881cd6de58b12cfcce9ef253402c5c2c94abc4bb22c19c007634d0d025668"} Jul 24 23:34:19.484776 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.484753 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" event={"ID":"9f4517be-0e39-481b-9428-0fcad57eb605","Type":"ContainerStarted","Data":"af8b4ebc3689ae268690440e5ab6f5034a14dd8698ad2717cd5e1547355e7b61"} Jul 24 23:34:19.484877 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.484838 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:19.518348 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.518311 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6d7d7d465-f2xmv" podStartSLOduration=1.948790383 podStartE2EDuration="5.518297941s" totalImagesPullingTime="3.569507558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:19.514845553 +0000 UTC m=+319.207686396" watchObservedRunningTime="2026-07-24 23:34:19.518297941 +0000 UTC m=+319.211138782" Jul 24 23:34:19.550910 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:19.550852 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" podStartSLOduration=1.492554816 podStartE2EDuration="4.55083346s" totalImagesPullingTime="3.058278644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:19.549095047 +0000 UTC m=+319.241935876" watchObservedRunningTime="2026-07-24 23:34:19.55083346 +0000 UTC m=+319.243674290" Jul 24 23:34:20.491215 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:20.491184 2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6zjr9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 23:34:20.491576 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:20.491253 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 23:34:22.486715 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.486676 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-qksm2"] Jul 24 23:34:22.490464 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.490430 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:22.499424 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.499276 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 23:34:22.502174 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.502150 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8"] Jul 24 23:34:22.503258 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.503218 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qv4bb\"" Jul 24 23:34:22.506024 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.505990 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-qksm2"] Jul 24 23:34:22.506145 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.506095 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:22.508525 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.508501 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 23:34:22.508704 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.508688 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mdzn4\"" Jul 24 23:34:22.521296 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.520961 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8"] Jul 24 23:34:22.521726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.521695 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddfaabf-761b-4973-a628-64aed03c455b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:22.521846 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.521767 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srbqx\" (UniqueName: \"kubernetes.io/projected/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-kube-api-access-srbqx\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:22.521908 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.521859 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpmq\" (UniqueName: \"kubernetes.io/projected/6ddfaabf-761b-4973-a628-64aed03c455b-kube-api-access-fcpmq\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:22.521963 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.521906 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-cert\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:22.622859 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.622822 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpmq\" (UniqueName: \"kubernetes.io/projected/6ddfaabf-761b-4973-a628-64aed03c455b-kube-api-access-fcpmq\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:22.623064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.622897 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-cert\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:22.623064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.622937 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddfaabf-761b-4973-a628-64aed03c455b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:22.623064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.622974 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srbqx\" (UniqueName: \"kubernetes.io/projected/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-kube-api-access-srbqx\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:22.623280 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:22.623144 2586 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 24 23:34:22.623280 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:22.623219 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-cert podName:8b8cb50c-23ae-404d-8e75-95b7bd9ca504 nodeName:}" failed. No retries permitted until 2026-07-24 23:34:23.123198406 +0000 UTC m=+322.816039229 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-cert") pod "kserve-controller-manager-d6f4b65b8-qksm2" (UID: "8b8cb50c-23ae-404d-8e75-95b7bd9ca504") : secret "kserve-webhook-server-cert" not found Jul 24 23:34:22.623537 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:22.623507 2586 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 24 23:34:22.623654 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:34:22.623575 2586 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ddfaabf-761b-4973-a628-64aed03c455b-cert podName:6ddfaabf-761b-4973-a628-64aed03c455b nodeName:}" failed. No retries permitted until 2026-07-24 23:34:23.123560157 +0000 UTC m=+322.816400964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/6ddfaabf-761b-4973-a628-64aed03c455b-cert") pod "llmisvc-controller-manager-777c6bd7d5-sn8x8" (UID: "6ddfaabf-761b-4973-a628-64aed03c455b") : secret "llmisvc-webhook-server-cert" not found Jul 24 23:34:22.633659 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.633614 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpmq\" (UniqueName: \"kubernetes.io/projected/6ddfaabf-761b-4973-a628-64aed03c455b-kube-api-access-fcpmq\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:22.633883 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:22.633643 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srbqx\" (UniqueName: \"kubernetes.io/projected/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-kube-api-access-srbqx\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:23.128446 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.128398 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-cert\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:23.128446 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.128454 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddfaabf-761b-4973-a628-64aed03c455b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:23.130963 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.130934 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b8cb50c-23ae-404d-8e75-95b7bd9ca504-cert\") pod \"kserve-controller-manager-d6f4b65b8-qksm2\" (UID: \"8b8cb50c-23ae-404d-8e75-95b7bd9ca504\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:23.131059 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.130941 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6ddfaabf-761b-4973-a628-64aed03c455b-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-sn8x8\" (UID: \"6ddfaabf-761b-4973-a628-64aed03c455b\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:23.404539 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.404445 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:23.419956 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.419924 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:23.490753 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.490620 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:23.572927 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.572824 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-qksm2"] Jul 24 23:34:23.575699 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:34:23.575662 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8cb50c_23ae_404d_8e75_95b7bd9ca504.slice/crio-6d2b925d10a72b13c739cfe137a811e213531e58b9424d62ba3918213f764418 WatchSource:0}: Error finding container 6d2b925d10a72b13c739cfe137a811e213531e58b9424d62ba3918213f764418: Status 404 returned error can't find the container with id 6d2b925d10a72b13c739cfe137a811e213531e58b9424d62ba3918213f764418 Jul 24 23:34:23.594503 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:23.594460 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8"] Jul 24 23:34:23.596951 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:34:23.596919 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6ddfaabf_761b_4973_a628_64aed03c455b.slice/crio-b9497191fa631ee1fc6d6d833e118e531270097f67b1554a7394aa1d120b63cd WatchSource:0}: Error finding container b9497191fa631ee1fc6d6d833e118e531270097f67b1554a7394aa1d120b63cd: Status 404 returned error can't find the container with id b9497191fa631ee1fc6d6d833e118e531270097f67b1554a7394aa1d120b63cd Jul 24 23:34:24.515545 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:24.515481 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" event={"ID":"6ddfaabf-761b-4973-a628-64aed03c455b","Type":"ContainerStarted","Data":"b9497191fa631ee1fc6d6d833e118e531270097f67b1554a7394aa1d120b63cd"} Jul 24 23:34:24.516948 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:24.516912 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" event={"ID":"8b8cb50c-23ae-404d-8e75-95b7bd9ca504","Type":"ContainerStarted","Data":"6d2b925d10a72b13c739cfe137a811e213531e58b9424d62ba3918213f764418"} Jul 24 23:34:25.608010 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:25.607848 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9"] Jul 24 23:34:25.608707 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:25.608658 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" containerName="discovery" containerID="cri-o://af8b4ebc3689ae268690440e5ab6f5034a14dd8698ad2717cd5e1547355e7b61" gracePeriod=30 Jul 24 23:34:26.466589 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.466555 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dx2nm" Jul 24 23:34:26.530857 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.530818 2586 generic.go:356] "Generic (PLEG): container finished" podID="9f4517be-0e39-481b-9428-0fcad57eb605" containerID="af8b4ebc3689ae268690440e5ab6f5034a14dd8698ad2717cd5e1547355e7b61" exitCode=0 Jul 24 23:34:26.531004 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.530891 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" event={"ID":"9f4517be-0e39-481b-9428-0fcad57eb605","Type":"ContainerDied","Data":"af8b4ebc3689ae268690440e5ab6f5034a14dd8698ad2717cd5e1547355e7b61"} Jul 24 23:34:26.531004 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.530954 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" event={"ID":"9f4517be-0e39-481b-9428-0fcad57eb605","Type":"ContainerDied","Data":"0eee0f43dc68bb82e731218a3e76d28749bb4ac65873b7f77ad67ff589c36686"} Jul 24 23:34:26.531004 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.530975 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eee0f43dc68bb82e731218a3e76d28749bb4ac65873b7f77ad67ff589c36686" Jul 24 23:34:26.532807 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.532784 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" event={"ID":"8b8cb50c-23ae-404d-8e75-95b7bd9ca504","Type":"ContainerStarted","Data":"09d36d9860ae3109d28ff2b62a99e2db1e0554741019e996e6e64cd33b3afab6"} Jul 24 23:34:26.532924 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.532881 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:26.543408 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.543382 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:26.562228 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.562178 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" podStartSLOduration=2.218651941 podStartE2EDuration="4.562158159s" totalImagesPullingTime="2.343506218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:26.561767632 +0000 UTC m=+326.254608461" watchObservedRunningTime="2026-07-24 23:34:26.562158159 +0000 UTC m=+326.254998988" Jul 24 23:34:26.661443 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661413 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-kubeconfig\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.661879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661453 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-cacerts\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.661879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661484 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-dns-cert\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.661879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661519 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjj4v\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-kube-api-access-jjj4v\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.661879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661556 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-istio-token\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.661879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661598 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4517be-0e39-481b-9428-0fcad57eb605-local-certs\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.661879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.661686 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-ca-configmap\") pod \"9f4517be-0e39-481b-9428-0fcad57eb605\" (UID: \"9f4517be-0e39-481b-9428-0fcad57eb605\") " Jul 24 23:34:26.662214 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.662189 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:34:26.664076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.664052 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-cacerts" (OuterVolumeSpecName: "cacerts") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:34:26.664280 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.664213 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4517be-0e39-481b-9428-0fcad57eb605-local-certs" (OuterVolumeSpecName: "local-certs") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:34:26.664363 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.664275 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-kube-api-access-jjj4v" (OuterVolumeSpecName: "kube-api-access-jjj4v") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "kube-api-access-jjj4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:34:26.664416 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.664370 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:34:26.664465 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.664401 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:34:26.664523 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.664503 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-istio-token" (OuterVolumeSpecName: "istio-token") pod "9f4517be-0e39-481b-9428-0fcad57eb605" (UID: "9f4517be-0e39-481b-9428-0fcad57eb605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:34:26.762740 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762650 2586 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-kubeconfig\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:26.762740 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762689 2586 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-cacerts\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:26.762740 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762703 2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-dns-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:26.762740 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762713 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjj4v\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-kube-api-access-jjj4v\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:26.762740 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762727 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9f4517be-0e39-481b-9428-0fcad57eb605-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:26.762740 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762739 2586 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9f4517be-0e39-481b-9428-0fcad57eb605-local-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:26.763046 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:26.762751 2586 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9f4517be-0e39-481b-9428-0fcad57eb605-istio-csr-ca-configmap\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:34:27.539014 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:27.538964 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" event={"ID":"6ddfaabf-761b-4973-a628-64aed03c455b","Type":"ContainerStarted","Data":"e468b670a09a4b5383c708468b6f1c56f7f5f70576881bcfc3aa99d0c43475eb"} Jul 24 23:34:27.539014 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:27.539005 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" Jul 24 23:34:27.539382 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:27.539365 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:34:27.567853 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:27.567790 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" podStartSLOduration=2.585887575 podStartE2EDuration="5.56777438s" totalImagesPullingTime="2.981886805s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:34:27.566108072 +0000 UTC m=+327.258948964" watchObservedRunningTime="2026-07-24 23:34:27.56777438 +0000 UTC m=+327.260615208" Jul 24 23:34:27.590293 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:27.590232 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9"] Jul 24 23:34:27.597315 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:27.597286 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9"] Jul 24 23:34:28.786289 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:28.786257 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" path="/var/lib/kubelet/pods/9f4517be-0e39-481b-9428-0fcad57eb605/volumes" Jul 24 23:34:31.490547 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:31.490504 2586 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6zjr9 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 23:34:31.490959 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:31.490587 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6zjr9" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.29:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 23:34:57.544249 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:57.544171 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-qksm2" Jul 24 23:34:59.551546 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:34:59.551516 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-sn8x8" Jul 24 23:35:12.807809 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.807773 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjwlw"] Jul 24 23:35:12.808364 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.808135 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" containerName="discovery" Jul 24 23:35:12.808364 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.808151 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" containerName="discovery" Jul 24 23:35:12.808364 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.808275 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f4517be-0e39-481b-9428-0fcad57eb605" containerName="discovery" Jul 24 23:35:12.810625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.810601 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:12.813915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.813892 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 23:35:12.814727 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.814694 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 23:35:12.814849 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.814701 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-b5w5s\"" Jul 24 23:35:12.822437 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.822415 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjwlw"] Jul 24 23:35:12.906145 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:12.906114 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q47z\" (UniqueName: \"kubernetes.io/projected/0ba1902c-ecd9-4a9f-b87a-9fc90aa74390-kube-api-access-4q47z\") pod \"kuadrant-operator-catalog-wjwlw\" (UID: \"0ba1902c-ecd9-4a9f-b87a-9fc90aa74390\") " pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:13.006936 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:13.006907 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q47z\" (UniqueName: \"kubernetes.io/projected/0ba1902c-ecd9-4a9f-b87a-9fc90aa74390-kube-api-access-4q47z\") pod \"kuadrant-operator-catalog-wjwlw\" (UID: \"0ba1902c-ecd9-4a9f-b87a-9fc90aa74390\") " pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:13.014649 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:13.014629 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q47z\" (UniqueName: \"kubernetes.io/projected/0ba1902c-ecd9-4a9f-b87a-9fc90aa74390-kube-api-access-4q47z\") pod \"kuadrant-operator-catalog-wjwlw\" (UID: \"0ba1902c-ecd9-4a9f-b87a-9fc90aa74390\") " pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:13.121410 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:13.121385 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:13.245997 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:13.245967 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wjwlw"] Jul 24 23:35:13.249720 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:35:13.249695 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba1902c_ecd9_4a9f_b87a_9fc90aa74390.slice/crio-b7eafe25898e7420a1ad1b92c23ab1de151439bb0c0fa656b683c8eb123c21be WatchSource:0}: Error finding container b7eafe25898e7420a1ad1b92c23ab1de151439bb0c0fa656b683c8eb123c21be: Status 404 returned error can't find the container with id b7eafe25898e7420a1ad1b92c23ab1de151439bb0c0fa656b683c8eb123c21be Jul 24 23:35:13.744785 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:13.744750 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" event={"ID":"0ba1902c-ecd9-4a9f-b87a-9fc90aa74390","Type":"ContainerStarted","Data":"b7eafe25898e7420a1ad1b92c23ab1de151439bb0c0fa656b683c8eb123c21be"} Jul 24 23:35:15.755900 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:15.755853 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" event={"ID":"0ba1902c-ecd9-4a9f-b87a-9fc90aa74390","Type":"ContainerStarted","Data":"cefe9aa4a6ac4116573db1fca2f1cb21f908921ed266ae75dff3dbdeb2bad57d"} Jul 24 23:35:15.773816 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:15.773767 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" podStartSLOduration=1.891062517 podStartE2EDuration="3.773752904s" totalImagesPullingTime="1.882690387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:35:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:35:15.7716468 +0000 UTC m=+375.464487629" watchObservedRunningTime="2026-07-24 23:35:15.773752904 +0000 UTC m=+375.466593732" Jul 24 23:35:23.121533 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:23.121485 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:23.122026 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:23.121559 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:23.143399 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:23.143372 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:23.816000 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:23.815968 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-wjwlw" Jul 24 23:35:47.551189 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.551151 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm"] Jul 24 23:35:47.554657 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.554637 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:35:47.557089 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.557061 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gdrt5\"" Jul 24 23:35:47.557214 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.557105 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 23:35:47.563121 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.563095 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm"] Jul 24 23:35:47.593373 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.593339 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/d08a8ae1-982c-4a7f-9629-cfecc6d58f87-kube-api-access-qcfmg\") pod \"dns-operator-controller-manager-864dd78c4c-5bxmm\" (UID: \"d08a8ae1-982c-4a7f-9629-cfecc6d58f87\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:35:47.693891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.693853 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/d08a8ae1-982c-4a7f-9629-cfecc6d58f87-kube-api-access-qcfmg\") pod \"dns-operator-controller-manager-864dd78c4c-5bxmm\" (UID: \"d08a8ae1-982c-4a7f-9629-cfecc6d58f87\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:35:47.708388 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.708352 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcfmg\" (UniqueName: \"kubernetes.io/projected/d08a8ae1-982c-4a7f-9629-cfecc6d58f87-kube-api-access-qcfmg\") pod \"dns-operator-controller-manager-864dd78c4c-5bxmm\" (UID: \"d08a8ae1-982c-4a7f-9629-cfecc6d58f87\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:35:47.864846 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:47.864804 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:35:48.003348 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:48.003322 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm"] Jul 24 23:35:48.005664 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:35:48.005636 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd08a8ae1_982c_4a7f_9629_cfecc6d58f87.slice/crio-60e9ac770aaa136a10130dc6ba52fd6c4d1390ec33a24142a27d70a67a31d257 WatchSource:0}: Error finding container 60e9ac770aaa136a10130dc6ba52fd6c4d1390ec33a24142a27d70a67a31d257: Status 404 returned error can't find the container with id 60e9ac770aaa136a10130dc6ba52fd6c4d1390ec33a24142a27d70a67a31d257 Jul 24 23:35:48.907907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:48.907868 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" event={"ID":"d08a8ae1-982c-4a7f-9629-cfecc6d58f87","Type":"ContainerStarted","Data":"60e9ac770aaa136a10130dc6ba52fd6c4d1390ec33a24142a27d70a67a31d257"} Jul 24 23:35:50.919084 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:50.919052 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" event={"ID":"d08a8ae1-982c-4a7f-9629-cfecc6d58f87","Type":"ContainerStarted","Data":"4a85344f725835cceabfb438b2f7ca505049d407ca40deb388fcd7b5fe5b7c06"} Jul 24 23:35:50.919516 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:50.919115 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:35:50.937925 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:50.937879 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" podStartSLOduration=2.0831846 podStartE2EDuration="3.93786585s" totalImagesPullingTime="1.85468125s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:35:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:35:50.935690737 +0000 UTC m=+410.628531564" watchObservedRunningTime="2026-07-24 23:35:50.93786585 +0000 UTC m=+410.630706678" Jul 24 23:35:52.646460 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.646424 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn"] Jul 24 23:35:52.649885 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.649862 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.652395 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.652379 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9p65l\"" Jul 24 23:35:52.665401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.665371 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn"] Jul 24 23:35:52.730281 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.730251 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgm6l\" (UniqueName: \"kubernetes.io/projected/b8b2915d-401f-4422-ac82-07adb75a9dc2-kube-api-access-jgm6l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.730390 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.730336 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b2915d-401f-4422-ac82-07adb75a9dc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.830915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.830892 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgm6l\" (UniqueName: \"kubernetes.io/projected/b8b2915d-401f-4422-ac82-07adb75a9dc2-kube-api-access-jgm6l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.831016 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.830956 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b2915d-401f-4422-ac82-07adb75a9dc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.831313 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.831298 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b2915d-401f-4422-ac82-07adb75a9dc2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.841915 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.841893 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgm6l\" (UniqueName: \"kubernetes.io/projected/b8b2915d-401f-4422-ac82-07adb75a9dc2-kube-api-access-jgm6l\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:52.960028 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:52.959976 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:53.094651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:53.094610 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn"] Jul 24 23:35:53.098804 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:35:53.098771 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b2915d_401f_4422_ac82_07adb75a9dc2.slice/crio-16e0f9f9662e7998d0aa7ce0997efdf68c5984583cc325316e348f2d93585ff9 WatchSource:0}: Error finding container 16e0f9f9662e7998d0aa7ce0997efdf68c5984583cc325316e348f2d93585ff9: Status 404 returned error can't find the container with id 16e0f9f9662e7998d0aa7ce0997efdf68c5984583cc325316e348f2d93585ff9 Jul 24 23:35:53.934526 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:53.934455 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" event={"ID":"b8b2915d-401f-4422-ac82-07adb75a9dc2","Type":"ContainerStarted","Data":"16e0f9f9662e7998d0aa7ce0997efdf68c5984583cc325316e348f2d93585ff9"} Jul 24 23:35:56.267255 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.267209 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p"] Jul 24 23:35:56.270765 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.270743 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:35:56.273432 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.273407 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-l7mmg\"" Jul 24 23:35:56.283633 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.283609 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p"] Jul 24 23:35:56.358572 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.358539 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfbj4\" (UniqueName: \"kubernetes.io/projected/ecbc90ba-0969-41a6-8517-aeb74095097e-kube-api-access-zfbj4\") pod \"authorino-operator-75cc8d5cb7-7rx8p\" (UID: \"ecbc90ba-0969-41a6-8517-aeb74095097e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:35:56.458995 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.458960 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfbj4\" (UniqueName: \"kubernetes.io/projected/ecbc90ba-0969-41a6-8517-aeb74095097e-kube-api-access-zfbj4\") pod \"authorino-operator-75cc8d5cb7-7rx8p\" (UID: \"ecbc90ba-0969-41a6-8517-aeb74095097e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:35:56.470265 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.470226 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfbj4\" (UniqueName: \"kubernetes.io/projected/ecbc90ba-0969-41a6-8517-aeb74095097e-kube-api-access-zfbj4\") pod \"authorino-operator-75cc8d5cb7-7rx8p\" (UID: \"ecbc90ba-0969-41a6-8517-aeb74095097e\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:35:56.583493 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:56.583425 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:35:57.612806 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:57.612783 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p"] Jul 24 23:35:57.615552 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:35:57.615526 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbc90ba_0969_41a6_8517_aeb74095097e.slice/crio-7b651e38be7686b3dae81d69303f277486adf0b6176403f33c00247d3ae177aa WatchSource:0}: Error finding container 7b651e38be7686b3dae81d69303f277486adf0b6176403f33c00247d3ae177aa: Status 404 returned error can't find the container with id 7b651e38be7686b3dae81d69303f277486adf0b6176403f33c00247d3ae177aa Jul 24 23:35:57.957311 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:57.957278 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" event={"ID":"b8b2915d-401f-4422-ac82-07adb75a9dc2","Type":"ContainerStarted","Data":"25f11299b2a90626e75ae0583f807dea0c5f02220ebfa2841e7027bbe5441095"} Jul 24 23:35:57.957496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:57.957449 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:35:57.958557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:57.958527 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" event={"ID":"ecbc90ba-0969-41a6-8517-aeb74095097e","Type":"ContainerStarted","Data":"7b651e38be7686b3dae81d69303f277486adf0b6176403f33c00247d3ae177aa"} Jul 24 23:35:57.976576 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:57.976532 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" podStartSLOduration=1.53263104 podStartE2EDuration="5.976515511s" totalImagesPullingTime="4.443884471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:35:57.974100987 +0000 UTC m=+417.666941810" watchObservedRunningTime="2026-07-24 23:35:57.976515511 +0000 UTC m=+417.669356338" Jul 24 23:35:59.717483 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.717449 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d"] Jul 24 23:35:59.726374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.726352 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:35:59.729001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.728977 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-x28rt\"" Jul 24 23:35:59.732307 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.732285 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d"] Jul 24 23:35:59.787064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.787034 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kkhd\" (UniqueName: \"kubernetes.io/projected/2c999e9c-d1df-4790-9970-4736d134e296-kube-api-access-2kkhd\") pod \"limitador-operator-controller-manager-859d7b78bd-r896d\" (UID: \"2c999e9c-d1df-4790-9970-4736d134e296\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:35:59.887949 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.887916 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kkhd\" (UniqueName: \"kubernetes.io/projected/2c999e9c-d1df-4790-9970-4736d134e296-kube-api-access-2kkhd\") pod \"limitador-operator-controller-manager-859d7b78bd-r896d\" (UID: \"2c999e9c-d1df-4790-9970-4736d134e296\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:35:59.897040 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:35:59.897011 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kkhd\" (UniqueName: \"kubernetes.io/projected/2c999e9c-d1df-4790-9970-4736d134e296-kube-api-access-2kkhd\") pod \"limitador-operator-controller-manager-859d7b78bd-r896d\" (UID: \"2c999e9c-d1df-4790-9970-4736d134e296\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:00.040589 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:00.040510 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:00.502774 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:00.502749 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d"] Jul 24 23:36:00.504554 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:36:00.504528 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c999e9c_d1df_4790_9970_4736d134e296.slice/crio-148c3e657a6d2bc1ebc560f26057629be7bd2be7aa1a46f5fa09b5e35f1be17d WatchSource:0}: Error finding container 148c3e657a6d2bc1ebc560f26057629be7bd2be7aa1a46f5fa09b5e35f1be17d: Status 404 returned error can't find the container with id 148c3e657a6d2bc1ebc560f26057629be7bd2be7aa1a46f5fa09b5e35f1be17d Jul 24 23:36:00.976148 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:00.976112 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" event={"ID":"2c999e9c-d1df-4790-9970-4736d134e296","Type":"ContainerStarted","Data":"148c3e657a6d2bc1ebc560f26057629be7bd2be7aa1a46f5fa09b5e35f1be17d"} Jul 24 23:36:00.977631 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:00.977600 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" event={"ID":"ecbc90ba-0969-41a6-8517-aeb74095097e","Type":"ContainerStarted","Data":"8b3233077fc9a89f1f63dae7a3b81c682a10e4d6180c92f1797b667dd7743005"} Jul 24 23:36:00.977738 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:00.977725 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:36:01.004608 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:01.004567 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" podStartSLOduration=2.197107599 podStartE2EDuration="5.004554487s" totalImagesPullingTime="2.807446888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:35:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:36:01.003395564 +0000 UTC m=+420.696236390" watchObservedRunningTime="2026-07-24 23:36:01.004554487 +0000 UTC m=+420.697395314" Jul 24 23:36:01.926172 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:01.926142 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5bxmm" Jul 24 23:36:02.988602 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:02.988563 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" event={"ID":"2c999e9c-d1df-4790-9970-4736d134e296","Type":"ContainerStarted","Data":"60ff34faaab5365d1b5cbc17a126259b44f1184b2d16a97a775454e0d6cf4951"} Jul 24 23:36:02.988956 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:02.988645 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:03.003649 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:03.003605 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" podStartSLOduration=2.562205035 podStartE2EDuration="4.003590574s" totalImagesPullingTime="1.441385539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:36:03.002344194 +0000 UTC m=+422.695185022" watchObservedRunningTime="2026-07-24 23:36:03.003590574 +0000 UTC m=+422.696431401" Jul 24 23:36:08.965874 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:08.965843 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:10.870885 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.870848 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn"] Jul 24 23:36:10.871433 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.871031 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" containerName="manager" containerID="cri-o://25f11299b2a90626e75ae0583f807dea0c5f02220ebfa2841e7027bbe5441095" gracePeriod=2 Jul 24 23:36:10.877287 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.877264 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn"] Jul 24 23:36:10.897256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.896936 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d"] Jul 24 23:36:10.897256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.897134 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" podUID="2c999e9c-d1df-4790-9970-4736d134e296" containerName="manager" containerID="cri-o://60ff34faaab5365d1b5cbc17a126259b44f1184b2d16a97a775454e0d6cf4951" gracePeriod=2 Jul 24 23:36:10.898988 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.898966 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:10.899623 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.899601 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv"] Jul 24 23:36:10.900131 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.900112 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" containerName="manager" Jul 24 23:36:10.900224 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.900133 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" containerName="manager" Jul 24 23:36:10.900312 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.900269 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" containerName="manager" Jul 24 23:36:10.903272 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.903255 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:10.908229 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.908210 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d"] Jul 24 23:36:10.916451 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:10.916417 2586 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"2c999e9c-d1df-4790-9970-4736d134e296\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-r896d\": pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:10.918879 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:10.918851 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:10.919385 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.919364 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv"] Jul 24 23:36:10.920657 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:10.920633 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:10.932642 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.932610 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p"] Jul 24 23:36:10.932969 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.932952 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c999e9c-d1df-4790-9970-4736d134e296" containerName="manager" Jul 24 23:36:10.932969 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.932969 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c999e9c-d1df-4790-9970-4736d134e296" containerName="manager" Jul 24 23:36:10.933102 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.933036 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c999e9c-d1df-4790-9970-4736d134e296" containerName="manager" Jul 24 23:36:10.936028 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.935986 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:10.971642 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:10.971604 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:10.972983 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.972955 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p"] Jul 24 23:36:10.973469 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:10.973442 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:10.976805 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.976781 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dgq7\" (UniqueName: \"kubernetes.io/projected/79060740-4a78-4bfe-a642-2271705f8576-kube-api-access-2dgq7\") pod \"limitador-operator-controller-manager-859d7b78bd-fbt6p\" (UID: \"79060740-4a78-4bfe-a642-2271705f8576\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:10.976902 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.976822 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15bae2ac-d3da-45ca-959c-3a565592c33c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8fzhv\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:10.976902 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:10.976875 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp5hm\" (UniqueName: \"kubernetes.io/projected/15bae2ac-d3da-45ca-959c-3a565592c33c-kube-api-access-wp5hm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8fzhv\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:10.992568 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:10.992543 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:11.033432 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.033400 2586 generic.go:356] "Generic (PLEG): container finished" podID="2c999e9c-d1df-4790-9970-4736d134e296" containerID="60ff34faaab5365d1b5cbc17a126259b44f1184b2d16a97a775454e0d6cf4951" exitCode=0 Jul 24 23:36:11.036054 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.036026 2586 generic.go:356] "Generic (PLEG): container finished" podID="b8b2915d-401f-4422-ac82-07adb75a9dc2" containerID="25f11299b2a90626e75ae0583f807dea0c5f02220ebfa2841e7027bbe5441095" exitCode=0 Jul 24 23:36:11.077841 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.077815 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp5hm\" (UniqueName: \"kubernetes.io/projected/15bae2ac-d3da-45ca-959c-3a565592c33c-kube-api-access-wp5hm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8fzhv\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:11.077991 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.077878 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgq7\" (UniqueName: \"kubernetes.io/projected/79060740-4a78-4bfe-a642-2271705f8576-kube-api-access-2dgq7\") pod \"limitador-operator-controller-manager-859d7b78bd-fbt6p\" (UID: \"79060740-4a78-4bfe-a642-2271705f8576\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:11.077991 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.077912 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15bae2ac-d3da-45ca-959c-3a565592c33c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8fzhv\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:11.078544 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.078341 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15bae2ac-d3da-45ca-959c-3a565592c33c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8fzhv\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:11.086058 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.086034 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp5hm\" (UniqueName: \"kubernetes.io/projected/15bae2ac-d3da-45ca-959c-3a565592c33c-kube-api-access-wp5hm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-8fzhv\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:11.086360 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.086338 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgq7\" (UniqueName: \"kubernetes.io/projected/79060740-4a78-4bfe-a642-2271705f8576-kube-api-access-2dgq7\") pod \"limitador-operator-controller-manager-859d7b78bd-fbt6p\" (UID: \"79060740-4a78-4bfe-a642-2271705f8576\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:11.129163 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.129142 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:11.131172 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.131147 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:11.133014 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.132992 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:11.135099 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.135083 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:11.137654 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.137634 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:11.139487 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.139466 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:11.179022 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.179006 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jgm6l\" (UniqueName: \"kubernetes.io/projected/b8b2915d-401f-4422-ac82-07adb75a9dc2-kube-api-access-jgm6l\") pod \"b8b2915d-401f-4422-ac82-07adb75a9dc2\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " Jul 24 23:36:11.179103 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.179032 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b2915d-401f-4422-ac82-07adb75a9dc2-extensions-socket-volume\") pod \"b8b2915d-401f-4422-ac82-07adb75a9dc2\" (UID: \"b8b2915d-401f-4422-ac82-07adb75a9dc2\") " Jul 24 23:36:11.179103 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.179049 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kkhd\" (UniqueName: \"kubernetes.io/projected/2c999e9c-d1df-4790-9970-4736d134e296-kube-api-access-2kkhd\") pod \"2c999e9c-d1df-4790-9970-4736d134e296\" (UID: \"2c999e9c-d1df-4790-9970-4736d134e296\") " Jul 24 23:36:11.179437 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.179412 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b2915d-401f-4422-ac82-07adb75a9dc2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b8b2915d-401f-4422-ac82-07adb75a9dc2" (UID: "b8b2915d-401f-4422-ac82-07adb75a9dc2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:36:11.181128 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.181106 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b2915d-401f-4422-ac82-07adb75a9dc2-kube-api-access-jgm6l" (OuterVolumeSpecName: "kube-api-access-jgm6l") pod "b8b2915d-401f-4422-ac82-07adb75a9dc2" (UID: "b8b2915d-401f-4422-ac82-07adb75a9dc2"). InnerVolumeSpecName "kube-api-access-jgm6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:36:11.181201 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.181124 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c999e9c-d1df-4790-9970-4736d134e296-kube-api-access-2kkhd" (OuterVolumeSpecName: "kube-api-access-2kkhd") pod "2c999e9c-d1df-4790-9970-4736d134e296" (UID: "2c999e9c-d1df-4790-9970-4736d134e296"). InnerVolumeSpecName "kube-api-access-2kkhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:36:11.279891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.279868 2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b8b2915d-401f-4422-ac82-07adb75a9dc2-extensions-socket-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:36:11.279891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.279890 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kkhd\" (UniqueName: \"kubernetes.io/projected/2c999e9c-d1df-4790-9970-4736d134e296-kube-api-access-2kkhd\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:36:11.280090 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.279899 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgm6l\" (UniqueName: \"kubernetes.io/projected/b8b2915d-401f-4422-ac82-07adb75a9dc2-kube-api-access-jgm6l\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:36:11.304139 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.304113 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:11.312139 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.312117 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:11.457936 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.457901 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv"] Jul 24 23:36:11.461808 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:36:11.461772 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bae2ac_d3da_45ca_959c_3a565592c33c.slice/crio-19f5ecf63f81ad6c1a3f227120dcc56b3b4c7fcf124183fc957e4c16dc7e77cf WatchSource:0}: Error finding container 19f5ecf63f81ad6c1a3f227120dcc56b3b4c7fcf124183fc957e4c16dc7e77cf: Status 404 returned error can't find the container with id 19f5ecf63f81ad6c1a3f227120dcc56b3b4c7fcf124183fc957e4c16dc7e77cf Jul 24 23:36:11.473161 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.473136 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p"] Jul 24 23:36:11.474800 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:36:11.474778 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79060740_4a78_4bfe_a642_2271705f8576.slice/crio-7856837ac6dd6125f1627f95af5a7247f519edbe3f79a4467c7e73549dd55157 WatchSource:0}: Error finding container 7856837ac6dd6125f1627f95af5a7247f519edbe3f79a4467c7e73549dd55157: Status 404 returned error can't find the container with id 7856837ac6dd6125f1627f95af5a7247f519edbe3f79a4467c7e73549dd55157 Jul 24 23:36:11.929541 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.929511 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh"] Jul 24 23:36:11.932978 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.932960 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:11.937218 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.937185 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:11.939131 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.939099 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:11.949185 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.949160 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh"] Jul 24 23:36:11.984920 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.984889 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d85801cb-37ed-4f13-8696-131a2d939e05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q7hmh\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:11.985024 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.984948 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4mjv\" (UniqueName: \"kubernetes.io/projected/d85801cb-37ed-4f13-8696-131a2d939e05-kube-api-access-b4mjv\") pod \"kuadrant-operator-controller-manager-86878c769f-q7hmh\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:11.985274 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:11.985221 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-7rx8p" Jul 24 23:36:11.987346 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:11.987324 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:12.007986 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:12.007952 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:12.042272 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.042215 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" event={"ID":"15bae2ac-d3da-45ca-959c-3a565592c33c","Type":"ContainerStarted","Data":"f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8"} Jul 24 23:36:12.042457 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.042292 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" event={"ID":"15bae2ac-d3da-45ca-959c-3a565592c33c","Type":"ContainerStarted","Data":"19f5ecf63f81ad6c1a3f227120dcc56b3b4c7fcf124183fc957e4c16dc7e77cf"} Jul 24 23:36:12.042457 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.042424 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:12.043710 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.043691 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:12.043816 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.043691 2586 scope.go:117] "RemoveContainer" containerID="60ff34faaab5365d1b5cbc17a126259b44f1184b2d16a97a775454e0d6cf4951" Jul 24 23:36:12.044652 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:12.044621 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:12.045610 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.045556 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" event={"ID":"79060740-4a78-4bfe-a642-2271705f8576","Type":"ContainerStarted","Data":"4d36cfe1a195495fddcde4a16221ce4f50460719bed832910c7203d1ee3f7897"} Jul 24 23:36:12.045610 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.045600 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" event={"ID":"79060740-4a78-4bfe-a642-2271705f8576","Type":"ContainerStarted","Data":"7856837ac6dd6125f1627f95af5a7247f519edbe3f79a4467c7e73549dd55157"} Jul 24 23:36:12.045796 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.045633 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:12.047114 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.047091 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:12.055090 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.055068 2586 scope.go:117] "RemoveContainer" containerID="25f11299b2a90626e75ae0583f807dea0c5f02220ebfa2841e7027bbe5441095" Jul 24 23:36:12.074342 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.074292 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" podStartSLOduration=2.074277216 podStartE2EDuration="2.074277216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:36:12.073419658 +0000 UTC m=+431.766260486" watchObservedRunningTime="2026-07-24 23:36:12.074277216 +0000 UTC m=+431.767118042" Jul 24 23:36:12.075807 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:12.075777 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:12.077574 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:12.077551 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-mqvqn\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mqvqn" Jul 24 23:36:12.079309 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:12.079289 2586 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-r896d\" is forbidden: User \"system:node:ip-10-0-131-63.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-131-63.ec2.internal' and this object" podUID="2c999e9c-d1df-4790-9970-4736d134e296" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-r896d" Jul 24 23:36:12.085390 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.085367 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4mjv\" (UniqueName: \"kubernetes.io/projected/d85801cb-37ed-4f13-8696-131a2d939e05-kube-api-access-b4mjv\") pod \"kuadrant-operator-controller-manager-86878c769f-q7hmh\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:12.085544 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.085512 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d85801cb-37ed-4f13-8696-131a2d939e05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q7hmh\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:12.085866 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.085846 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d85801cb-37ed-4f13-8696-131a2d939e05-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-q7hmh\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:12.094806 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.094787 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4mjv\" (UniqueName: \"kubernetes.io/projected/d85801cb-37ed-4f13-8696-131a2d939e05-kube-api-access-b4mjv\") pod \"kuadrant-operator-controller-manager-86878c769f-q7hmh\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:12.101643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.101598 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" podStartSLOduration=2.101579723 podStartE2EDuration="2.101579723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:36:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:36:12.099373998 +0000 UTC m=+431.792214824" watchObservedRunningTime="2026-07-24 23:36:12.101579723 +0000 UTC m=+431.794420551" Jul 24 23:36:12.244908 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.244825 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:12.385608 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.385578 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh"] Jul 24 23:36:12.388928 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:36:12.388899 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85801cb_37ed_4f13_8696_131a2d939e05.slice/crio-9ccadfef0e35e6b6b92eefa62936ae17438e3f8b63f629b19cf9d687d80e4f7e WatchSource:0}: Error finding container 9ccadfef0e35e6b6b92eefa62936ae17438e3f8b63f629b19cf9d687d80e4f7e: Status 404 returned error can't find the container with id 9ccadfef0e35e6b6b92eefa62936ae17438e3f8b63f629b19cf9d687d80e4f7e Jul 24 23:36:12.787453 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.787421 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c999e9c-d1df-4790-9970-4736d134e296" path="/var/lib/kubelet/pods/2c999e9c-d1df-4790-9970-4736d134e296/volumes" Jul 24 23:36:12.787800 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:12.787786 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b8b2915d-401f-4422-ac82-07adb75a9dc2" path="/var/lib/kubelet/pods/b8b2915d-401f-4422-ac82-07adb75a9dc2/volumes" Jul 24 23:36:13.053121 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:13.053022 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" event={"ID":"d85801cb-37ed-4f13-8696-131a2d939e05","Type":"ContainerStarted","Data":"c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83"} Jul 24 23:36:13.053559 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:13.053166 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:13.053559 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:13.053192 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" event={"ID":"d85801cb-37ed-4f13-8696-131a2d939e05","Type":"ContainerStarted","Data":"9ccadfef0e35e6b6b92eefa62936ae17438e3f8b63f629b19cf9d687d80e4f7e"} Jul 24 23:36:13.098878 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:13.098823 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" podStartSLOduration=2.098808827 podStartE2EDuration="2.098808827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:36:13.09630472 +0000 UTC m=+432.789145559" watchObservedRunningTime="2026-07-24 23:36:13.098808827 +0000 UTC m=+432.791649652" Jul 24 23:36:23.056936 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:23.056900 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:23.057529 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:23.057034 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-fbt6p" Jul 24 23:36:24.062660 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.062626 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:36:24.116776 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.116746 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv"] Jul 24 23:36:24.117008 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.116960 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" podUID="15bae2ac-d3da-45ca-959c-3a565592c33c" containerName="manager" containerID="cri-o://f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8" gracePeriod=10 Jul 24 23:36:24.382914 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.382884 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:24.495144 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.495105 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15bae2ac-d3da-45ca-959c-3a565592c33c-extensions-socket-volume\") pod \"15bae2ac-d3da-45ca-959c-3a565592c33c\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " Jul 24 23:36:24.495368 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.495214 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wp5hm\" (UniqueName: \"kubernetes.io/projected/15bae2ac-d3da-45ca-959c-3a565592c33c-kube-api-access-wp5hm\") pod \"15bae2ac-d3da-45ca-959c-3a565592c33c\" (UID: \"15bae2ac-d3da-45ca-959c-3a565592c33c\") " Jul 24 23:36:24.495569 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.495542 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15bae2ac-d3da-45ca-959c-3a565592c33c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "15bae2ac-d3da-45ca-959c-3a565592c33c" (UID: "15bae2ac-d3da-45ca-959c-3a565592c33c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:36:24.497304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.497279 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15bae2ac-d3da-45ca-959c-3a565592c33c-kube-api-access-wp5hm" (OuterVolumeSpecName: "kube-api-access-wp5hm") pod "15bae2ac-d3da-45ca-959c-3a565592c33c" (UID: "15bae2ac-d3da-45ca-959c-3a565592c33c"). InnerVolumeSpecName "kube-api-access-wp5hm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:36:24.596661 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.596573 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp5hm\" (UniqueName: \"kubernetes.io/projected/15bae2ac-d3da-45ca-959c-3a565592c33c-kube-api-access-wp5hm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:36:24.596661 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:24.596605 2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/15bae2ac-d3da-45ca-959c-3a565592c33c-extensions-socket-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:36:25.127217 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.127184 2586 generic.go:356] "Generic (PLEG): container finished" podID="15bae2ac-d3da-45ca-959c-3a565592c33c" containerID="f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8" exitCode=0 Jul 24 23:36:25.127643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.127261 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" Jul 24 23:36:25.127643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.127269 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" event={"ID":"15bae2ac-d3da-45ca-959c-3a565592c33c","Type":"ContainerDied","Data":"f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8"} Jul 24 23:36:25.127643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.127330 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv" event={"ID":"15bae2ac-d3da-45ca-959c-3a565592c33c","Type":"ContainerDied","Data":"19f5ecf63f81ad6c1a3f227120dcc56b3b4c7fcf124183fc957e4c16dc7e77cf"} Jul 24 23:36:25.127643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.127361 2586 scope.go:117] "RemoveContainer" containerID="f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8" Jul 24 23:36:25.135617 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.135594 2586 scope.go:117] "RemoveContainer" containerID="f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8" Jul 24 23:36:25.135874 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:25.135855 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8\": container with ID starting with f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8 not found: ID does not exist" containerID="f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8" Jul 24 23:36:25.135932 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.135887 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8"} err="failed to get container status \"f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8\": rpc error: code = NotFound desc = could not find container \"f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8\": container with ID starting with f135054dac6a31ba8d1ad3e99eddb2fd9b886d4fcf7e1c97ff67f26eeb1ea5e8 not found: ID does not exist" Jul 24 23:36:25.145113 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.145087 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv"] Jul 24 23:36:25.150730 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:25.150709 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-8fzhv"] Jul 24 23:36:26.787408 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:26.787376 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15bae2ac-d3da-45ca-959c-3a565592c33c" path="/var/lib/kubelet/pods/15bae2ac-d3da-45ca-959c-3a565592c33c/volumes" Jul 24 23:36:43.051816 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.051776 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nsz"] Jul 24 23:36:43.052284 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.052203 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15bae2ac-d3da-45ca-959c-3a565592c33c" containerName="manager" Jul 24 23:36:43.052284 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.052216 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="15bae2ac-d3da-45ca-959c-3a565592c33c" containerName="manager" Jul 24 23:36:43.052399 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.052291 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="15bae2ac-d3da-45ca-959c-3a565592c33c" containerName="manager" Jul 24 23:36:43.057306 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.057269 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:43.060122 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.060091 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d59d4\"" Jul 24 23:36:43.062922 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.062900 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nsz"] Jul 24 23:36:43.150222 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.150188 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdqrj\" (UniqueName: \"kubernetes.io/projected/04a9bf26-7f29-4635-bd15-09674fb61f91-kube-api-access-qdqrj\") pod \"authorino-58cd955675-x8nsz\" (UID: \"04a9bf26-7f29-4635-bd15-09674fb61f91\") " pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:43.250935 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.250896 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdqrj\" (UniqueName: \"kubernetes.io/projected/04a9bf26-7f29-4635-bd15-09674fb61f91-kube-api-access-qdqrj\") pod \"authorino-58cd955675-x8nsz\" (UID: \"04a9bf26-7f29-4635-bd15-09674fb61f91\") " pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:43.259937 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.259896 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdqrj\" (UniqueName: \"kubernetes.io/projected/04a9bf26-7f29-4635-bd15-09674fb61f91-kube-api-access-qdqrj\") pod \"authorino-58cd955675-x8nsz\" (UID: \"04a9bf26-7f29-4635-bd15-09674fb61f91\") " pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:43.367433 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.367399 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:43.501066 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:43.501033 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nsz"] Jul 24 23:36:43.503325 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:36:43.503294 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04a9bf26_7f29_4635_bd15_09674fb61f91.slice/crio-fd1c0849a00ca7225b81f7ff4bc85f8cc1974d825a1557dbae0864ac571ef69b WatchSource:0}: Error finding container fd1c0849a00ca7225b81f7ff4bc85f8cc1974d825a1557dbae0864ac571ef69b: Status 404 returned error can't find the container with id fd1c0849a00ca7225b81f7ff4bc85f8cc1974d825a1557dbae0864ac571ef69b Jul 24 23:36:44.223212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:44.223176 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nsz" event={"ID":"04a9bf26-7f29-4635-bd15-09674fb61f91","Type":"ContainerStarted","Data":"fd1c0849a00ca7225b81f7ff4bc85f8cc1974d825a1557dbae0864ac571ef69b"} Jul 24 23:36:47.243903 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:47.243871 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nsz" event={"ID":"04a9bf26-7f29-4635-bd15-09674fb61f91","Type":"ContainerStarted","Data":"d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43"} Jul 24 23:36:47.258064 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:47.258024 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-x8nsz" podStartSLOduration=1.056304082 podStartE2EDuration="4.25801087s" totalImagesPullingTime="3.201706788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:36:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:36:47.256978783 +0000 UTC m=+466.949819617" watchObservedRunningTime="2026-07-24 23:36:47.25801087 +0000 UTC m=+466.950851696" Jul 24 23:36:48.327310 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:48.327274 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nsz"] Jul 24 23:36:49.253840 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:49.253796 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-x8nsz" podUID="04a9bf26-7f29-4635-bd15-09674fb61f91" containerName="authorino" containerID="cri-o://d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43" gracePeriod=30 Jul 24 23:36:49.495112 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:49.495084 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:49.604018 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:49.603995 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdqrj\" (UniqueName: \"kubernetes.io/projected/04a9bf26-7f29-4635-bd15-09674fb61f91-kube-api-access-qdqrj\") pod \"04a9bf26-7f29-4635-bd15-09674fb61f91\" (UID: \"04a9bf26-7f29-4635-bd15-09674fb61f91\") " Jul 24 23:36:49.605891 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:49.605869 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a9bf26-7f29-4635-bd15-09674fb61f91-kube-api-access-qdqrj" (OuterVolumeSpecName: "kube-api-access-qdqrj") pod "04a9bf26-7f29-4635-bd15-09674fb61f91" (UID: "04a9bf26-7f29-4635-bd15-09674fb61f91"). InnerVolumeSpecName "kube-api-access-qdqrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:36:49.704665 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:49.704642 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdqrj\" (UniqueName: \"kubernetes.io/projected/04a9bf26-7f29-4635-bd15-09674fb61f91-kube-api-access-qdqrj\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:36:50.260322 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.260288 2586 generic.go:356] "Generic (PLEG): container finished" podID="04a9bf26-7f29-4635-bd15-09674fb61f91" containerID="d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43" exitCode=0 Jul 24 23:36:50.260464 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.260333 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-x8nsz" Jul 24 23:36:50.260464 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.260378 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nsz" event={"ID":"04a9bf26-7f29-4635-bd15-09674fb61f91","Type":"ContainerDied","Data":"d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43"} Jul 24 23:36:50.260464 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.260420 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-x8nsz" event={"ID":"04a9bf26-7f29-4635-bd15-09674fb61f91","Type":"ContainerDied","Data":"fd1c0849a00ca7225b81f7ff4bc85f8cc1974d825a1557dbae0864ac571ef69b"} Jul 24 23:36:50.260464 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.260439 2586 scope.go:117] "RemoveContainer" containerID="d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43" Jul 24 23:36:50.269873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.269856 2586 scope.go:117] "RemoveContainer" containerID="d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43" Jul 24 23:36:50.270120 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:36:50.270100 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43\": container with ID starting with d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43 not found: ID does not exist" containerID="d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43" Jul 24 23:36:50.270213 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.270125 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43"} err="failed to get container status \"d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43\": rpc error: code = NotFound desc = could not find container \"d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43\": container with ID starting with d4a4337bda38edf84d9be8272d1ab356b80c471a3a6c85ff5da68f9cb900ca43 not found: ID does not exist" Jul 24 23:36:50.282103 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.282081 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nsz"] Jul 24 23:36:50.285764 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.285745 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-x8nsz"] Jul 24 23:36:50.786546 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:36:50.786516 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a9bf26-7f29-4635-bd15-09674fb61f91" path="/var/lib/kubelet/pods/04a9bf26-7f29-4635-bd15-09674fb61f91/volumes" Jul 24 23:37:16.982441 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.982399 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-grq78"] Jul 24 23:37:16.982954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.982933 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a9bf26-7f29-4635-bd15-09674fb61f91" containerName="authorino" Jul 24 23:37:16.983031 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.982956 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a9bf26-7f29-4635-bd15-09674fb61f91" containerName="authorino" Jul 24 23:37:16.983086 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.983077 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a9bf26-7f29-4635-bd15-09674fb61f91" containerName="authorino" Jul 24 23:37:16.987551 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.987531 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:16.989774 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.989755 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-d59d4\"" Jul 24 23:37:16.992725 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:16.992706 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-grq78"] Jul 24 23:37:17.014463 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.014440 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bphqj\" (UniqueName: \"kubernetes.io/projected/4d314548-9b45-45df-b20f-4cae1da79fc8-kube-api-access-bphqj\") pod \"authorino-5d4b67c446-grq78\" (UID: \"4d314548-9b45-45df-b20f-4cae1da79fc8\") " pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:17.114892 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.114860 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bphqj\" (UniqueName: \"kubernetes.io/projected/4d314548-9b45-45df-b20f-4cae1da79fc8-kube-api-access-bphqj\") pod \"authorino-5d4b67c446-grq78\" (UID: \"4d314548-9b45-45df-b20f-4cae1da79fc8\") " pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:17.127715 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.127692 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bphqj\" (UniqueName: \"kubernetes.io/projected/4d314548-9b45-45df-b20f-4cae1da79fc8-kube-api-access-bphqj\") pod \"authorino-5d4b67c446-grq78\" (UID: \"4d314548-9b45-45df-b20f-4cae1da79fc8\") " pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:17.148113 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.148087 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-grq78"] Jul 24 23:37:17.148377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.148361 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:17.172418 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.172397 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-qdbdg"] Jul 24 23:37:17.176818 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.176798 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.179221 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.179201 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 24 23:37:17.183367 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.183342 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-qdbdg"] Jul 24 23:37:17.216136 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.216100 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e717137-7a96-4410-97ec-ebc02b6ca64f-tls-cert\") pod \"authorino-76877994-qdbdg\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.216324 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.216160 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkdtv\" (UniqueName: \"kubernetes.io/projected/6e717137-7a96-4410-97ec-ebc02b6ca64f-kube-api-access-wkdtv\") pod \"authorino-76877994-qdbdg\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.243584 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.243549 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-qdbdg"] Jul 24 23:37:17.243836 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:37:17.243811 2586 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wkdtv tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-qdbdg" podUID="6e717137-7a96-4410-97ec-ebc02b6ca64f" Jul 24 23:37:17.269688 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.269663 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-9575c8547-hg727"] Jul 24 23:37:17.272873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.272856 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.281652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.281628 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9575c8547-hg727"] Jul 24 23:37:17.287389 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.287371 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-grq78"] Jul 24 23:37:17.290327 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:37:17.290296 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d314548_9b45_45df_b20f_4cae1da79fc8.slice/crio-2cb32e53b6deb4e1327690bdcb3a2c41c77591705b1f88703c238e66fa73a681 WatchSource:0}: Error finding container 2cb32e53b6deb4e1327690bdcb3a2c41c77591705b1f88703c238e66fa73a681: Status 404 returned error can't find the container with id 2cb32e53b6deb4e1327690bdcb3a2c41c77591705b1f88703c238e66fa73a681 Jul 24 23:37:17.317285 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.317223 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dktb8\" (UniqueName: \"kubernetes.io/projected/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-kube-api-access-dktb8\") pod \"authorino-9575c8547-hg727\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.317374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.317295 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e717137-7a96-4410-97ec-ebc02b6ca64f-tls-cert\") pod \"authorino-76877994-qdbdg\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.317374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.317325 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-tls-cert\") pod \"authorino-9575c8547-hg727\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.317374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.317353 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkdtv\" (UniqueName: \"kubernetes.io/projected/6e717137-7a96-4410-97ec-ebc02b6ca64f-kube-api-access-wkdtv\") pod \"authorino-76877994-qdbdg\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.319436 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.319418 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e717137-7a96-4410-97ec-ebc02b6ca64f-tls-cert\") pod \"authorino-76877994-qdbdg\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.325226 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.325197 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkdtv\" (UniqueName: \"kubernetes.io/projected/6e717137-7a96-4410-97ec-ebc02b6ca64f-kube-api-access-wkdtv\") pod \"authorino-76877994-qdbdg\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.387011 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.386967 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-grq78" event={"ID":"4d314548-9b45-45df-b20f-4cae1da79fc8","Type":"ContainerStarted","Data":"2cb32e53b6deb4e1327690bdcb3a2c41c77591705b1f88703c238e66fa73a681"} Jul 24 23:37:17.387182 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.387037 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.394547 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.394525 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:17.417754 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.417728 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkdtv\" (UniqueName: \"kubernetes.io/projected/6e717137-7a96-4410-97ec-ebc02b6ca64f-kube-api-access-wkdtv\") pod \"6e717137-7a96-4410-97ec-ebc02b6ca64f\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " Jul 24 23:37:17.417855 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.417809 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e717137-7a96-4410-97ec-ebc02b6ca64f-tls-cert\") pod \"6e717137-7a96-4410-97ec-ebc02b6ca64f\" (UID: \"6e717137-7a96-4410-97ec-ebc02b6ca64f\") " Jul 24 23:37:17.417960 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.417942 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dktb8\" (UniqueName: \"kubernetes.io/projected/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-kube-api-access-dktb8\") pod \"authorino-9575c8547-hg727\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.418015 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.417978 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-tls-cert\") pod \"authorino-9575c8547-hg727\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.419797 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.419774 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e717137-7a96-4410-97ec-ebc02b6ca64f-kube-api-access-wkdtv" (OuterVolumeSpecName: "kube-api-access-wkdtv") pod "6e717137-7a96-4410-97ec-ebc02b6ca64f" (UID: "6e717137-7a96-4410-97ec-ebc02b6ca64f"). InnerVolumeSpecName "kube-api-access-wkdtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:37:17.419981 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.419958 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e717137-7a96-4410-97ec-ebc02b6ca64f-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "6e717137-7a96-4410-97ec-ebc02b6ca64f" (UID: "6e717137-7a96-4410-97ec-ebc02b6ca64f"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:37:17.420415 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.420401 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-tls-cert\") pod \"authorino-9575c8547-hg727\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.425377 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.425352 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dktb8\" (UniqueName: \"kubernetes.io/projected/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-kube-api-access-dktb8\") pod \"authorino-9575c8547-hg727\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.518907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.518832 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkdtv\" (UniqueName: \"kubernetes.io/projected/6e717137-7a96-4410-97ec-ebc02b6ca64f-kube-api-access-wkdtv\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.518907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.518862 2586 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6e717137-7a96-4410-97ec-ebc02b6ca64f-tls-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:37:17.582723 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.582703 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:37:17.724560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:17.724532 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-9575c8547-hg727"] Jul 24 23:37:17.726908 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:37:17.726882 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b9d5a84_07ef_480b_ab7f_9f5fcc818c3c.slice/crio-00a39934782f88f10abb1d02d6fd81ca6affdc75331442f55342f6140718a987 WatchSource:0}: Error finding container 00a39934782f88f10abb1d02d6fd81ca6affdc75331442f55342f6140718a987: Status 404 returned error can't find the container with id 00a39934782f88f10abb1d02d6fd81ca6affdc75331442f55342f6140718a987 Jul 24 23:37:18.393807 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.393772 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-grq78" event={"ID":"4d314548-9b45-45df-b20f-4cae1da79fc8","Type":"ContainerStarted","Data":"c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd"} Jul 24 23:37:18.394219 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.393842 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-grq78" podUID="4d314548-9b45-45df-b20f-4cae1da79fc8" containerName="authorino" containerID="cri-o://c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd" gracePeriod=30 Jul 24 23:37:18.395499 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.395472 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9575c8547-hg727" event={"ID":"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c","Type":"ContainerStarted","Data":"a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9"} Jul 24 23:37:18.395597 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.395512 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9575c8547-hg727" event={"ID":"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c","Type":"ContainerStarted","Data":"00a39934782f88f10abb1d02d6fd81ca6affdc75331442f55342f6140718a987"} Jul 24 23:37:18.395597 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.395527 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-qdbdg" Jul 24 23:37:18.408130 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.408089 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-grq78" podStartSLOduration=2.138468907 podStartE2EDuration="2.408073589s" totalImagesPullingTime="269.604682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:37:18.406767341 +0000 UTC m=+498.099608158" watchObservedRunningTime="2026-07-24 23:37:18.408073589 +0000 UTC m=+498.100914417" Jul 24 23:37:18.424561 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.424475 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-9575c8547-hg727" podStartSLOduration=1.085841754 podStartE2EDuration="1.424458697s" totalImagesPullingTime="338.616943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:37:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:37:18.424134919 +0000 UTC m=+498.116975746" watchObservedRunningTime="2026-07-24 23:37:18.424458697 +0000 UTC m=+498.117299526" Jul 24 23:37:18.488418 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.488382 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-qdbdg"] Jul 24 23:37:18.491671 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.491641 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-qdbdg"] Jul 24 23:37:18.648927 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.648860 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:18.730284 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.730247 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bphqj\" (UniqueName: \"kubernetes.io/projected/4d314548-9b45-45df-b20f-4cae1da79fc8-kube-api-access-bphqj\") pod \"4d314548-9b45-45df-b20f-4cae1da79fc8\" (UID: \"4d314548-9b45-45df-b20f-4cae1da79fc8\") " Jul 24 23:37:18.732552 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.732523 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d314548-9b45-45df-b20f-4cae1da79fc8-kube-api-access-bphqj" (OuterVolumeSpecName: "kube-api-access-bphqj") pod "4d314548-9b45-45df-b20f-4cae1da79fc8" (UID: "4d314548-9b45-45df-b20f-4cae1da79fc8"). InnerVolumeSpecName "kube-api-access-bphqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:37:18.787324 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.787291 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e717137-7a96-4410-97ec-ebc02b6ca64f" path="/var/lib/kubelet/pods/6e717137-7a96-4410-97ec-ebc02b6ca64f/volumes" Jul 24 23:37:18.831297 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:18.831251 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bphqj\" (UniqueName: \"kubernetes.io/projected/4d314548-9b45-45df-b20f-4cae1da79fc8-kube-api-access-bphqj\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:37:19.401315 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.401277 2586 generic.go:356] "Generic (PLEG): container finished" podID="4d314548-9b45-45df-b20f-4cae1da79fc8" containerID="c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd" exitCode=0 Jul 24 23:37:19.401315 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.401316 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-grq78" Jul 24 23:37:19.401726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.401343 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-grq78" event={"ID":"4d314548-9b45-45df-b20f-4cae1da79fc8","Type":"ContainerDied","Data":"c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd"} Jul 24 23:37:19.401726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.401389 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-grq78" event={"ID":"4d314548-9b45-45df-b20f-4cae1da79fc8","Type":"ContainerDied","Data":"2cb32e53b6deb4e1327690bdcb3a2c41c77591705b1f88703c238e66fa73a681"} Jul 24 23:37:19.401726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.401407 2586 scope.go:117] "RemoveContainer" containerID="c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd" Jul 24 23:37:19.409652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.409637 2586 scope.go:117] "RemoveContainer" containerID="c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd" Jul 24 23:37:19.409891 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:37:19.409873 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd\": container with ID starting with c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd not found: ID does not exist" containerID="c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd" Jul 24 23:37:19.409951 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.409898 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd"} err="failed to get container status \"c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd\": rpc error: code = NotFound desc = could not find container \"c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd\": container with ID starting with c1eb97408e2d278560ef54e34eebec7f29b6cca8880ada8d7794b12b9aaed1bd not found: ID does not exist" Jul 24 23:37:19.415692 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.415673 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-grq78"] Jul 24 23:37:19.418889 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:19.418870 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-grq78"] Jul 24 23:37:20.786284 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:20.786232 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d314548-9b45-45df-b20f-4cae1da79fc8" path="/var/lib/kubelet/pods/4d314548-9b45-45df-b20f-4cae1da79fc8/volumes" Jul 24 23:37:59.063757 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.063686 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m"] Jul 24 23:37:59.064257 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.064048 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d314548-9b45-45df-b20f-4cae1da79fc8" containerName="authorino" Jul 24 23:37:59.064257 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.064059 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d314548-9b45-45df-b20f-4cae1da79fc8" containerName="authorino" Jul 24 23:37:59.064257 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.064137 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d314548-9b45-45df-b20f-4cae1da79fc8" containerName="authorino" Jul 24 23:37:59.067402 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.067381 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.069716 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.069696 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tjx4m\"" Jul 24 23:37:59.070060 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.070035 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 23:37:59.081056 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.081036 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m"] Jul 24 23:37:59.166726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.166702 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbfa169-1d5f-492b-b938-94f02db53897-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-dkr6m\" (UID: \"0dbfa169-1d5f-492b-b938-94f02db53897\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.166852 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.166750 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5k4\" (UniqueName: \"kubernetes.io/projected/0dbfa169-1d5f-492b-b938-94f02db53897-kube-api-access-jw5k4\") pod \"payload-pre-processing-687dbb6cf6-dkr6m\" (UID: \"0dbfa169-1d5f-492b-b938-94f02db53897\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.267520 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.267492 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5k4\" (UniqueName: \"kubernetes.io/projected/0dbfa169-1d5f-492b-b938-94f02db53897-kube-api-access-jw5k4\") pod \"payload-pre-processing-687dbb6cf6-dkr6m\" (UID: \"0dbfa169-1d5f-492b-b938-94f02db53897\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.267651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.267563 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbfa169-1d5f-492b-b938-94f02db53897-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-dkr6m\" (UID: \"0dbfa169-1d5f-492b-b938-94f02db53897\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.268416 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.268397 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0dbfa169-1d5f-492b-b938-94f02db53897-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-dkr6m\" (UID: \"0dbfa169-1d5f-492b-b938-94f02db53897\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.274894 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.274874 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5k4\" (UniqueName: \"kubernetes.io/projected/0dbfa169-1d5f-492b-b938-94f02db53897-kube-api-access-jw5k4\") pod \"payload-pre-processing-687dbb6cf6-dkr6m\" (UID: \"0dbfa169-1d5f-492b-b938-94f02db53897\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.379420 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.379399 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:37:59.718865 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:37:59.718840 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m"] Jul 24 23:37:59.721118 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:37:59.721089 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dbfa169_1d5f_492b_b938_94f02db53897.slice/crio-0d5419304047906844ef60605d5adc33e81a50f0cd249117af02688e7098ad0b WatchSource:0}: Error finding container 0d5419304047906844ef60605d5adc33e81a50f0cd249117af02688e7098ad0b: Status 404 returned error can't find the container with id 0d5419304047906844ef60605d5adc33e81a50f0cd249117af02688e7098ad0b Jul 24 23:38:00.609727 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:00.609692 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" event={"ID":"0dbfa169-1d5f-492b-b938-94f02db53897","Type":"ContainerStarted","Data":"0d5419304047906844ef60605d5adc33e81a50f0cd249117af02688e7098ad0b"} Jul 24 23:38:02.621183 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:02.621143 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" event={"ID":"0dbfa169-1d5f-492b-b938-94f02db53897","Type":"ContainerStarted","Data":"1cb5ba65816a4f67c9e1f7ba54ae82a1e469f2295da6adf9528c208d78b3625a"} Jul 24 23:38:02.621573 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:02.621223 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:38:02.637086 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:02.637037 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" podStartSLOduration=1.5313080000000001 podStartE2EDuration="3.637022367s" totalImagesPullingTime="2.105714367s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:37:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:02.635710242 +0000 UTC m=+542.328551069" watchObservedRunningTime="2026-07-24 23:38:02.637022367 +0000 UTC m=+542.329863194" Jul 24 23:38:13.629095 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:13.629063 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-dkr6m" Jul 24 23:38:15.284354 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.284324 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt"] Jul 24 23:38:15.288198 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.288178 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.290476 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.290456 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 23:38:15.290565 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.290460 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 23:38:15.290565 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.290461 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 24 23:38:15.291219 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.291204 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-n27r8\"" Jul 24 23:38:15.297571 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.297550 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt"] Jul 24 23:38:15.399601 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.399566 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.399601 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.399600 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.399819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.399623 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbnfg\" (UniqueName: \"kubernetes.io/projected/7fc943ed-3253-4673-8702-062707fde29f-kube-api-access-bbnfg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.399819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.399723 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc943ed-3253-4673-8702-062707fde29f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.399819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.399750 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.399819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.399768 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500500 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500470 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500500 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500501 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500702 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500526 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbnfg\" (UniqueName: \"kubernetes.io/projected/7fc943ed-3253-4673-8702-062707fde29f-kube-api-access-bbnfg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500702 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500662 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc943ed-3253-4673-8702-062707fde29f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500818 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500716 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500818 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500772 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.500936 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.500892 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.501063 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.501041 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.501157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.501139 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.502672 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.502653 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc943ed-3253-4673-8702-062707fde29f-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.502922 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.502907 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc943ed-3253-4673-8702-062707fde29f-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.507771 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.507744 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbnfg\" (UniqueName: \"kubernetes.io/projected/7fc943ed-3253-4673-8702-062707fde29f-kube-api-access-bbnfg\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt\" (UID: \"7fc943ed-3253-4673-8702-062707fde29f\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.599794 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.599736 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:15.738812 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:15.738781 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt"] Jul 24 23:38:15.742451 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:38:15.742420 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc943ed_3253_4673_8702_062707fde29f.slice/crio-5564861c9387a45b4de50a8df43f30ec7a3d8e2f14833cf6caccf223be8e2322 WatchSource:0}: Error finding container 5564861c9387a45b4de50a8df43f30ec7a3d8e2f14833cf6caccf223be8e2322: Status 404 returned error can't find the container with id 5564861c9387a45b4de50a8df43f30ec7a3d8e2f14833cf6caccf223be8e2322 Jul 24 23:38:16.690051 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:16.690017 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" event={"ID":"7fc943ed-3253-4673-8702-062707fde29f","Type":"ContainerStarted","Data":"5564861c9387a45b4de50a8df43f30ec7a3d8e2f14833cf6caccf223be8e2322"} Jul 24 23:38:17.696763 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:17.696725 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" event={"ID":"7fc943ed-3253-4673-8702-062707fde29f","Type":"ContainerStarted","Data":"7419a4b7f1c73fab6c5992d6ac4c6c1f2cf8bfa761811250571ecdbf9f6820e8"} Jul 24 23:38:17.713401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:17.713349 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" podStartSLOduration=1.556453284 podStartE2EDuration="2.713334231s" totalImagesPullingTime="1.156880947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:17.712209438 +0000 UTC m=+557.405050266" watchObservedRunningTime="2026-07-24 23:38:17.713334231 +0000 UTC m=+557.406175054" Jul 24 23:38:25.600715 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:25.600678 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:25.601185 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:25.600830 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:25.613395 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:25.613363 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:25.752168 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:25.752135 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt" Jul 24 23:38:47.490881 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.490839 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql"] Jul 24 23:38:47.496223 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.496200 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.498559 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.498529 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 23:38:47.504408 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.504380 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql"] Jul 24 23:38:47.578465 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.578423 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.578646 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.578475 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.578646 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.578566 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xqm6\" (UniqueName: \"kubernetes.io/projected/76a05693-1be0-4b6d-9969-dc78215c15b4-kube-api-access-8xqm6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.578646 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.578635 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.578750 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.578657 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.578750 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.578694 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a05693-1be0-4b6d-9969-dc78215c15b4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679436 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679398 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679436 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679438 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679539 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a05693-1be0-4b6d-9969-dc78215c15b4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679601 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679643 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679630 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679773 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679678 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xqm6\" (UniqueName: \"kubernetes.io/projected/76a05693-1be0-4b6d-9969-dc78215c15b4-kube-api-access-8xqm6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.679957 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679932 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.680078 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.679955 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.680078 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.680011 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.681700 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.681673 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76a05693-1be0-4b6d-9969-dc78215c15b4-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.682066 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.682045 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76a05693-1be0-4b6d-9969-dc78215c15b4-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.686851 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.686830 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xqm6\" (UniqueName: \"kubernetes.io/projected/76a05693-1be0-4b6d-9969-dc78215c15b4-kube-api-access-8xqm6\") pod \"e2e-trlp-test-simulated-kserve-598466548d-zgbql\" (UID: \"76a05693-1be0-4b6d-9969-dc78215c15b4\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.808399 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.808305 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:47.944980 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:47.944952 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql"] Jul 24 23:38:47.947014 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:38:47.946987 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76a05693_1be0_4b6d_9969_dc78215c15b4.slice/crio-28e189bdfbec1d6ca05ca2e4f0a798fd077c5fe3d317143d8a346e5650e54cdd WatchSource:0}: Error finding container 28e189bdfbec1d6ca05ca2e4f0a798fd077c5fe3d317143d8a346e5650e54cdd: Status 404 returned error can't find the container with id 28e189bdfbec1d6ca05ca2e4f0a798fd077c5fe3d317143d8a346e5650e54cdd Jul 24 23:38:48.848846 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:48.848803 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" event={"ID":"76a05693-1be0-4b6d-9969-dc78215c15b4","Type":"ContainerStarted","Data":"5c4c8857eef1e907897ad02ece5164a2b1bfe84cd9c7adc4da0a793b8ec8b7c6"} Jul 24 23:38:48.849433 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:48.849402 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" event={"ID":"76a05693-1be0-4b6d-9969-dc78215c15b4","Type":"ContainerStarted","Data":"28e189bdfbec1d6ca05ca2e4f0a798fd077c5fe3d317143d8a346e5650e54cdd"} Jul 24 23:38:48.868157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:48.868104 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" podStartSLOduration=1.652800447 podStartE2EDuration="1.868086335s" totalImagesPullingTime="215.285888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:38:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:38:48.866705459 +0000 UTC m=+588.559546285" watchObservedRunningTime="2026-07-24 23:38:48.868086335 +0000 UTC m=+588.560927161" Jul 24 23:38:57.808968 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:57.808929 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:57.809436 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:57.809012 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:57.820807 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:57.820788 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:38:57.905477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:38:57.905453 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-zgbql" Jul 24 23:39:00.760454 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:39:00.760425 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:39:00.763793 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:39:00.763771 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:41:00.799867 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:00.799801 2586 scope.go:117] "RemoveContainer" containerID="af8b4ebc3689ae268690440e5ab6f5034a14dd8698ad2717cd5e1547355e7b61" Jul 24 23:41:02.480712 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.480665 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9575c8547-hg727"] Jul 24 23:41:02.481216 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.480908 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-9575c8547-hg727" podUID="4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" containerName="authorino" containerID="cri-o://a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9" gracePeriod=30 Jul 24 23:41:02.731968 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.731908 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:41:02.817481 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.817453 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dktb8\" (UniqueName: \"kubernetes.io/projected/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-kube-api-access-dktb8\") pod \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " Jul 24 23:41:02.817661 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.817500 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-tls-cert\") pod \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\" (UID: \"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c\") " Jul 24 23:41:02.819540 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.819508 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-kube-api-access-dktb8" (OuterVolumeSpecName: "kube-api-access-dktb8") pod "4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" (UID: "4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c"). InnerVolumeSpecName "kube-api-access-dktb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:41:02.827713 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.827683 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" (UID: "4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 23:41:02.918392 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.918360 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dktb8\" (UniqueName: \"kubernetes.io/projected/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-kube-api-access-dktb8\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:41:02.918392 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:02.918388 2586 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c-tls-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:41:03.519001 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.518969 2586 generic.go:356] "Generic (PLEG): container finished" podID="4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" containerID="a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9" exitCode=0 Jul 24 23:41:03.519374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.519013 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-9575c8547-hg727" Jul 24 23:41:03.519374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.519056 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9575c8547-hg727" event={"ID":"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c","Type":"ContainerDied","Data":"a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9"} Jul 24 23:41:03.519374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.519102 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-9575c8547-hg727" event={"ID":"4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c","Type":"ContainerDied","Data":"00a39934782f88f10abb1d02d6fd81ca6affdc75331442f55342f6140718a987"} Jul 24 23:41:03.519374 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.519122 2586 scope.go:117] "RemoveContainer" containerID="a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9" Jul 24 23:41:03.530000 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.529983 2586 scope.go:117] "RemoveContainer" containerID="a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9" Jul 24 23:41:03.530562 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:41:03.530541 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9\": container with ID starting with a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9 not found: ID does not exist" containerID="a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9" Jul 24 23:41:03.530648 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.530573 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9"} err="failed to get container status \"a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9\": rpc error: code = NotFound desc = could not find container \"a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9\": container with ID starting with a5452d510ae245a6aed55754395b208b784416487211981ec8e3de82c45822f9 not found: ID does not exist" Jul 24 23:41:03.539347 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.539324 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-9575c8547-hg727"] Jul 24 23:41:03.541115 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:03.541095 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-9575c8547-hg727"] Jul 24 23:41:04.786231 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:41:04.786203 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" path="/var/lib/kubelet/pods/4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c/volumes" Jul 24 23:44:00.799463 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:44:00.799389 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:44:00.799983 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:44:00.799966 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:45:00.140399 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.140359 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn"] Jul 24 23:45:00.142517 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.140941 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" containerName="authorino" Jul 24 23:45:00.142517 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.140958 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" containerName="authorino" Jul 24 23:45:00.142517 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.141055 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b9d5a84-07ef-480b-ab7f-9f5fcc818c3c" containerName="authorino" Jul 24 23:45:00.144244 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.144218 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" Jul 24 23:45:00.146485 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.146465 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bxr46\"" Jul 24 23:45:00.154102 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.154081 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" Jul 24 23:45:00.155346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.155322 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn"] Jul 24 23:45:00.491871 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.491848 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn"] Jul 24 23:45:00.493856 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:45:00.493832 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53d03129_4ee9_468b_847f_3dc95de27a0d.slice/crio-7588124cbf37b6e5bc3fbe39bb3505e5d5fe0ca6e546c191cb75770338b2cfd3 WatchSource:0}: Error finding container 7588124cbf37b6e5bc3fbe39bb3505e5d5fe0ca6e546c191cb75770338b2cfd3: Status 404 returned error can't find the container with id 7588124cbf37b6e5bc3fbe39bb3505e5d5fe0ca6e546c191cb75770338b2cfd3 Jul 24 23:45:00.495705 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.495687 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:45:00.661070 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:00.661040 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" event={"ID":"53d03129-4ee9-468b-847f-3dc95de27a0d","Type":"ContainerStarted","Data":"7588124cbf37b6e5bc3fbe39bb3505e5d5fe0ca6e546c191cb75770338b2cfd3"} Jul 24 23:45:03.678703 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:03.678668 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" event={"ID":"53d03129-4ee9-468b-847f-3dc95de27a0d","Type":"ContainerStarted","Data":"2604927cabfa6075fe6cec38e8138282fbce765faa01f1535f18c23e4b8c61b5"} Jul 24 23:45:03.692847 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:03.692802 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" podStartSLOduration=1.272885161 podStartE2EDuration="3.692787465s" totalImagesPullingTime="2.419902304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:45:03.692455867 +0000 UTC m=+963.385296694" watchObservedRunningTime="2026-07-24 23:45:03.692787465 +0000 UTC m=+963.385628293" Jul 24 23:45:04.685184 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:04.685150 2586 generic.go:356] "Generic (PLEG): container finished" podID="53d03129-4ee9-468b-847f-3dc95de27a0d" containerID="2604927cabfa6075fe6cec38e8138282fbce765faa01f1535f18c23e4b8c61b5" exitCode=0 Jul 24 23:45:04.685588 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:04.685275 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" event={"ID":"53d03129-4ee9-468b-847f-3dc95de27a0d","Type":"ContainerDied","Data":"2604927cabfa6075fe6cec38e8138282fbce765faa01f1535f18c23e4b8c61b5"} Jul 24 23:45:05.824687 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:05.824659 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" Jul 24 23:45:06.695709 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:06.695685 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" Jul 24 23:45:06.695871 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:06.695678 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748945-rb5bn" event={"ID":"53d03129-4ee9-468b-847f-3dc95de27a0d","Type":"ContainerDied","Data":"7588124cbf37b6e5bc3fbe39bb3505e5d5fe0ca6e546c191cb75770338b2cfd3"} Jul 24 23:45:06.695871 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:45:06.695811 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7588124cbf37b6e5bc3fbe39bb3505e5d5fe0ca6e546c191cb75770338b2cfd3" Jul 24 23:47:55.589139 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:55.589105 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh"] Jul 24 23:47:55.589652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:55.589308 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" podUID="d85801cb-37ed-4f13-8696-131a2d939e05" containerName="manager" containerID="cri-o://c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83" gracePeriod=10 Jul 24 23:47:58.237660 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.237635 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:47:58.338087 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.338012 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d85801cb-37ed-4f13-8696-131a2d939e05-extensions-socket-volume\") pod \"d85801cb-37ed-4f13-8696-131a2d939e05\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " Jul 24 23:47:58.338087 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.338086 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b4mjv\" (UniqueName: \"kubernetes.io/projected/d85801cb-37ed-4f13-8696-131a2d939e05-kube-api-access-b4mjv\") pod \"d85801cb-37ed-4f13-8696-131a2d939e05\" (UID: \"d85801cb-37ed-4f13-8696-131a2d939e05\") " Jul 24 23:47:58.338366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.338345 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d85801cb-37ed-4f13-8696-131a2d939e05-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d85801cb-37ed-4f13-8696-131a2d939e05" (UID: "d85801cb-37ed-4f13-8696-131a2d939e05"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:47:58.339888 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.339866 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85801cb-37ed-4f13-8696-131a2d939e05-kube-api-access-b4mjv" (OuterVolumeSpecName: "kube-api-access-b4mjv") pod "d85801cb-37ed-4f13-8696-131a2d939e05" (UID: "d85801cb-37ed-4f13-8696-131a2d939e05"). InnerVolumeSpecName "kube-api-access-b4mjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:47:58.438856 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.438835 2586 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d85801cb-37ed-4f13-8696-131a2d939e05-extensions-socket-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:47:58.438856 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.438855 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4mjv\" (UniqueName: \"kubernetes.io/projected/d85801cb-37ed-4f13-8696-131a2d939e05-kube-api-access-b4mjv\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:47:58.524962 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.524933 2586 generic.go:356] "Generic (PLEG): container finished" podID="d85801cb-37ed-4f13-8696-131a2d939e05" containerID="c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83" exitCode=0 Jul 24 23:47:58.525067 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.524990 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" Jul 24 23:47:58.525067 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.524983 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" event={"ID":"d85801cb-37ed-4f13-8696-131a2d939e05","Type":"ContainerDied","Data":"c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83"} Jul 24 23:47:58.525149 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.525131 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh" event={"ID":"d85801cb-37ed-4f13-8696-131a2d939e05","Type":"ContainerDied","Data":"9ccadfef0e35e6b6b92eefa62936ae17438e3f8b63f629b19cf9d687d80e4f7e"} Jul 24 23:47:58.525193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.525169 2586 scope.go:117] "RemoveContainer" containerID="c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83" Jul 24 23:47:58.535277 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.535258 2586 scope.go:117] "RemoveContainer" containerID="c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83" Jul 24 23:47:58.535514 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:47:58.535496 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83\": container with ID starting with c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83 not found: ID does not exist" containerID="c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83" Jul 24 23:47:58.535572 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.535522 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83"} err="failed to get container status \"c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83\": rpc error: code = NotFound desc = could not find container \"c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83\": container with ID starting with c642abd3a53b4a103e1ee8129ecaf798a07aae4d818aa27baef1ffb853288a83 not found: ID does not exist" Jul 24 23:47:58.546232 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.546209 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh"] Jul 24 23:47:58.549687 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.549668 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-q7hmh"] Jul 24 23:47:58.786457 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:47:58.786431 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d85801cb-37ed-4f13-8696-131a2d939e05" path="/var/lib/kubelet/pods/d85801cb-37ed-4f13-8696-131a2d939e05/volumes" Jul 24 23:48:56.750207 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750131 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98"] Jul 24 23:48:56.750651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750494 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53d03129-4ee9-468b-847f-3dc95de27a0d" containerName="cleanup" Jul 24 23:48:56.750651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750505 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="53d03129-4ee9-468b-847f-3dc95de27a0d" containerName="cleanup" Jul 24 23:48:56.750651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750523 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d85801cb-37ed-4f13-8696-131a2d939e05" containerName="manager" Jul 24 23:48:56.750651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750528 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d85801cb-37ed-4f13-8696-131a2d939e05" containerName="manager" Jul 24 23:48:56.750651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750584 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d85801cb-37ed-4f13-8696-131a2d939e05" containerName="manager" Jul 24 23:48:56.750651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.750596 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="53d03129-4ee9-468b-847f-3dc95de27a0d" containerName="cleanup" Jul 24 23:48:56.753556 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.753540 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:56.756192 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.756164 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-9p65l\"" Jul 24 23:48:56.766316 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.766293 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98"] Jul 24 23:48:56.928172 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.928142 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8zm98\" (UID: \"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:56.928347 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:56.928187 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n56q6\" (UniqueName: \"kubernetes.io/projected/dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a-kube-api-access-n56q6\") pod \"kuadrant-operator-controller-manager-86878c769f-8zm98\" (UID: \"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.029410 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.029347 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8zm98\" (UID: \"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.029410 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.029395 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n56q6\" (UniqueName: \"kubernetes.io/projected/dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a-kube-api-access-n56q6\") pod \"kuadrant-operator-controller-manager-86878c769f-8zm98\" (UID: \"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.029687 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.029665 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-8zm98\" (UID: \"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.037590 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.037567 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n56q6\" (UniqueName: \"kubernetes.io/projected/dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a-kube-api-access-n56q6\") pod \"kuadrant-operator-controller-manager-86878c769f-8zm98\" (UID: \"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.063477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.063457 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.195454 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.195430 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98"] Jul 24 23:48:57.197978 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:48:57.197942 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc328c13_43c0_4a82_8d2e_f15f0d2c5c7a.slice/crio-1dc6b3a1b260c7207f47f7805bb7461671dff073ca0a5ea5ae51a2da806d6eed WatchSource:0}: Error finding container 1dc6b3a1b260c7207f47f7805bb7461671dff073ca0a5ea5ae51a2da806d6eed: Status 404 returned error can't find the container with id 1dc6b3a1b260c7207f47f7805bb7461671dff073ca0a5ea5ae51a2da806d6eed Jul 24 23:48:57.813738 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.813703 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" event={"ID":"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a","Type":"ContainerStarted","Data":"ea34c69ad06d1a60e55f8a00ed7e6cdd3e04fe9515c5d57e4dffc56de2c9338d"} Jul 24 23:48:57.814103 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.813755 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" event={"ID":"dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a","Type":"ContainerStarted","Data":"1dc6b3a1b260c7207f47f7805bb7461671dff073ca0a5ea5ae51a2da806d6eed"} Jul 24 23:48:57.814103 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.813785 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:48:57.837451 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:48:57.837391 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" podStartSLOduration=1.837375003 podStartE2EDuration="1.837375003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:48:57.835004387 +0000 UTC m=+1197.527845214" watchObservedRunningTime="2026-07-24 23:48:57.837375003 +0000 UTC m=+1197.530215830" Jul 24 23:49:00.828588 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:49:00.828556 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:49:00.829731 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:49:00.829713 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:49:08.820760 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:49:08.820731 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-8zm98" Jul 24 23:54:00.856608 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:54:00.856581 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:54:00.858958 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:54:00.858929 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:56:00.227798 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.227715 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs"] Jul 24 23:56:00.231410 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.231393 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.233770 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.233750 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 23:56:00.233985 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.233972 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-90db14c6-openshift-default-dockercfg-wrp54\"" Jul 24 23:56:00.249732 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.249705 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs"] Jul 24 23:56:00.361572 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361545 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-envoy\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361601 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-podinfo\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361665 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istiod-ca-cert\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361695 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-certs\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361858 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361744 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-data\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361858 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361783 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-token\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361858 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361806 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-credential-socket\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361858 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361824 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-socket\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.361858 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.361852 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq664\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-kube-api-access-zq664\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463229 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463202 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istiod-ca-cert\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463267 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-certs\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463296 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-data\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463332 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-token\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463357 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-credential-socket\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463401 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463382 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-socket\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463658 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463423 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq664\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-kube-api-access-zq664\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463658 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463480 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-envoy\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463658 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463543 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-podinfo\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463822 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463685 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-certs\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463822 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463732 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-credential-socket\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463822 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463796 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-socket\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.463971 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463856 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-data\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.464031 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.463971 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istiod-ca-cert\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.465950 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.465927 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-envoy\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.466190 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.466170 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-podinfo\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.479494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.479442 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq664\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-kube-api-access-zq664\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.479774 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.479754 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-token\") pod \"e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.543660 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.543633 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:00.681149 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.681126 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs"] Jul 24 23:56:00.683424 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:56:00.683396 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cde3110_9aca_4b76_9ada_a9e75a7ab708.slice/crio-34810ca460ae48ee165f73ef9175d4dbecc6e34d42621cb80dcb416a200146e1 WatchSource:0}: Error finding container 34810ca460ae48ee165f73ef9175d4dbecc6e34d42621cb80dcb416a200146e1: Status 404 returned error can't find the container with id 34810ca460ae48ee165f73ef9175d4dbecc6e34d42621cb80dcb416a200146e1 Jul 24 23:56:00.685449 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.685433 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 23:56:00.881068 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:00.881042 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" event={"ID":"7cde3110-9aca-4b76-9ada-a9e75a7ab708","Type":"ContainerStarted","Data":"34810ca460ae48ee165f73ef9175d4dbecc6e34d42621cb80dcb416a200146e1"} Jul 24 23:56:03.424225 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:03.424190 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:03.424516 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:03.424286 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:03.424516 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:03.424314 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:03.898599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:03.898568 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" event={"ID":"7cde3110-9aca-4b76-9ada-a9e75a7ab708","Type":"ContainerStarted","Data":"e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f"} Jul 24 23:56:03.917670 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:03.917622 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" podStartSLOduration=1.1792772280000001 podStartE2EDuration="3.917608567s" totalImagesPullingTime="2.738331339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:56:03.916325957 +0000 UTC m=+1623.609166785" watchObservedRunningTime="2026-07-24 23:56:03.917608567 +0000 UTC m=+1623.610449393" Jul 24 23:56:04.544101 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:04.544062 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:04.548908 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:04.548882 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:04.904316 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:04.904283 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:04.905111 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:04.905092 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:06.118657 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.118625 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h"] Jul 24 23:56:06.122444 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.122426 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.125436 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.125411 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-90db14c6\"" Jul 24 23:56:06.136972 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.136916 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h"] Jul 24 23:56:06.214795 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.214765 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5cs\" (UniqueName: \"kubernetes.io/projected/73c500e2-aa65-4d5b-8571-6320da448d80-kube-api-access-hh5cs\") pod \"payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.214947 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.214824 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c500e2-aa65-4d5b-8571-6320da448d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.315479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.315445 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c500e2-aa65-4d5b-8571-6320da448d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.315657 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.315514 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5cs\" (UniqueName: \"kubernetes.io/projected/73c500e2-aa65-4d5b-8571-6320da448d80-kube-api-access-hh5cs\") pod \"payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.316210 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.316188 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c500e2-aa65-4d5b-8571-6320da448d80-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.323323 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.323293 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5cs\" (UniqueName: \"kubernetes.io/projected/73c500e2-aa65-4d5b-8571-6320da448d80-kube-api-access-hh5cs\") pod \"payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.433384 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.433329 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.568076 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.567980 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h"] Jul 24 23:56:06.570998 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:56:06.570970 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c500e2_aa65_4d5b_8571_6320da448d80.slice/crio-ba4773bf6b2e5b02e4b257509517aedaefbfa3183b12a4eaa95ac42a46b47b90 WatchSource:0}: Error finding container ba4773bf6b2e5b02e4b257509517aedaefbfa3183b12a4eaa95ac42a46b47b90: Status 404 returned error can't find the container with id ba4773bf6b2e5b02e4b257509517aedaefbfa3183b12a4eaa95ac42a46b47b90 Jul 24 23:56:06.915020 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.914980 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" event={"ID":"73c500e2-aa65-4d5b-8571-6320da448d80","Type":"ContainerStarted","Data":"3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2"} Jul 24 23:56:06.915295 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.915282 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:06.915375 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.915308 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" event={"ID":"73c500e2-aa65-4d5b-8571-6320da448d80","Type":"ContainerStarted","Data":"ba4773bf6b2e5b02e4b257509517aedaefbfa3183b12a4eaa95ac42a46b47b90"} Jul 24 23:56:06.929079 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:06.929036 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" podStartSLOduration=0.929020057 podStartE2EDuration="929.020057ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:56:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:56:06.928217708 +0000 UTC m=+1626.621058539" watchObservedRunningTime="2026-07-24 23:56:06.929020057 +0000 UTC m=+1626.621860887" Jul 24 23:56:18.927538 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.927510 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:18.988436 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.988407 2586 generic.go:356] "Generic (PLEG): container finished" podID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerID="e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f" exitCode=0 Jul 24 23:56:18.988572 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.988445 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" event={"ID":"7cde3110-9aca-4b76-9ada-a9e75a7ab708","Type":"ContainerDied","Data":"e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f"} Jul 24 23:56:18.988859 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.988842 2586 scope.go:117] "RemoveContainer" containerID="e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f" Jul 24 23:56:18.990978 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.990947 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:18.991077 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.991058 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:18.991128 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:18.991114 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:19.904690 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:19.904639 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:19.995060 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:19.995017 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" event={"ID":"7cde3110-9aca-4b76-9ada-a9e75a7ab708","Type":"ContainerStarted","Data":"290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507"} Jul 24 23:56:20.544416 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:20.544375 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:20.545935 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:20.545898 2586 patch_prober.go:28] interesting pod/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:56:20.546068 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:20.545960 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 24 23:56:21.545340 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:21.545297 2586 patch_prober.go:28] interesting pod/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:56:21.545818 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:21.545362 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 24 23:56:22.544625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:22.544573 2586 patch_prober.go:28] interesting pod/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:56:22.544897 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:22.544652 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 24 23:56:23.545143 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:23.545101 2586 patch_prober.go:28] interesting pod/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 24 23:56:23.546145 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:23.545728 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 24 23:56:24.548080 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:24.548027 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:24.548583 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:24.548565 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:24.549325 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:24.549305 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:30.840408 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:30.840373 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5"] Jul 24 23:56:30.850283 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:30.850254 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5"] Jul 24 23:56:30.850435 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:30.850373 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:30.852705 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:30.852682 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bxr46\"" Jul 24 23:56:30.852842 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:30.852709 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:56:30.946053 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:30.946009 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98a9e6-797c-4548-96c8-181473e27225-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5\" (UID: \"9a98a9e6-797c-4548-96c8-181473e27225\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:31.046792 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:31.046752 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98a9e6-797c-4548-96c8-181473e27225-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5\" (UID: \"9a98a9e6-797c-4548-96c8-181473e27225\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:31.047537 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:31.047501 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98a9e6-797c-4548-96c8-181473e27225-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5\" (UID: \"9a98a9e6-797c-4548-96c8-181473e27225\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:31.161905 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:31.161874 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:31.297492 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:31.297462 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5"] Jul 24 23:56:31.299462 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:56:31.299432 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a98a9e6_797c_4548_96c8_181473e27225.slice/crio-2ad94f4ffa8a9f3a7cf796985553b114285ab4a47597b9989978e6061ecc78d3 WatchSource:0}: Error finding container 2ad94f4ffa8a9f3a7cf796985553b114285ab4a47597b9989978e6061ecc78d3: Status 404 returned error can't find the container with id 2ad94f4ffa8a9f3a7cf796985553b114285ab4a47597b9989978e6061ecc78d3 Jul 24 23:56:32.057086 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:32.057053 2586 generic.go:356] "Generic (PLEG): container finished" podID="9a98a9e6-797c-4548-96c8-181473e27225" containerID="32555cfc3a9f71552f0c2a11f8f0b9a35bde85ffa273d39e227f8481c77b92f0" exitCode=0 Jul 24 23:56:32.057449 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:32.057138 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" event={"ID":"9a98a9e6-797c-4548-96c8-181473e27225","Type":"ContainerDied","Data":"32555cfc3a9f71552f0c2a11f8f0b9a35bde85ffa273d39e227f8481c77b92f0"} Jul 24 23:56:32.057449 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:32.057198 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" event={"ID":"9a98a9e6-797c-4548-96c8-181473e27225","Type":"ContainerStarted","Data":"2ad94f4ffa8a9f3a7cf796985553b114285ab4a47597b9989978e6061ecc78d3"} Jul 24 23:56:33.199585 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:33.199559 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:33.265808 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:33.265773 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98a9e6-797c-4548-96c8-181473e27225-maas-api-service-ca\") pod \"9a98a9e6-797c-4548-96c8-181473e27225\" (UID: \"9a98a9e6-797c-4548-96c8-181473e27225\") " Jul 24 23:56:33.266142 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:33.266118 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a98a9e6-797c-4548-96c8-181473e27225-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9a98a9e6-797c-4548-96c8-181473e27225" (UID: "9a98a9e6-797c-4548-96c8-181473e27225"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:56:33.367095 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:33.367057 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a98a9e6-797c-4548-96c8-181473e27225-maas-api-service-ca\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:34.067899 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:34.067874 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" Jul 24 23:56:34.068120 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:34.067911 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5" event={"ID":"9a98a9e6-797c-4548-96c8-181473e27225","Type":"ContainerDied","Data":"2ad94f4ffa8a9f3a7cf796985553b114285ab4a47597b9989978e6061ecc78d3"} Jul 24 23:56:34.068120 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:34.067965 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ad94f4ffa8a9f3a7cf796985553b114285ab4a47597b9989978e6061ecc78d3" Jul 24 23:56:41.301302 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.301255 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h"] Jul 24 23:56:41.301782 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.301498 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" podUID="73c500e2-aa65-4d5b-8571-6320da448d80" containerName="payload-pre-processing" containerID="cri-o://3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2" gracePeriod=30 Jul 24 23:56:41.572929 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.572895 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:41.638292 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.638261 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c500e2-aa65-4d5b-8571-6320da448d80-plugins-config-volume\") pod \"73c500e2-aa65-4d5b-8571-6320da448d80\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " Jul 24 23:56:41.638485 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.638313 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hh5cs\" (UniqueName: \"kubernetes.io/projected/73c500e2-aa65-4d5b-8571-6320da448d80-kube-api-access-hh5cs\") pod \"73c500e2-aa65-4d5b-8571-6320da448d80\" (UID: \"73c500e2-aa65-4d5b-8571-6320da448d80\") " Jul 24 23:56:41.638759 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.638728 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c500e2-aa65-4d5b-8571-6320da448d80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73c500e2-aa65-4d5b-8571-6320da448d80" (UID: "73c500e2-aa65-4d5b-8571-6320da448d80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:56:41.640487 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.640458 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c500e2-aa65-4d5b-8571-6320da448d80-kube-api-access-hh5cs" (OuterVolumeSpecName: "kube-api-access-hh5cs") pod "73c500e2-aa65-4d5b-8571-6320da448d80" (UID: "73c500e2-aa65-4d5b-8571-6320da448d80"). InnerVolumeSpecName "kube-api-access-hh5cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:56:41.739199 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.739140 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73c500e2-aa65-4d5b-8571-6320da448d80-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:41.739199 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:41.739187 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh5cs\" (UniqueName: \"kubernetes.io/projected/73c500e2-aa65-4d5b-8571-6320da448d80-kube-api-access-hh5cs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:42.112159 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.112119 2586 generic.go:356] "Generic (PLEG): container finished" podID="73c500e2-aa65-4d5b-8571-6320da448d80" containerID="3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2" exitCode=0 Jul 24 23:56:42.112373 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.112166 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" Jul 24 23:56:42.112373 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.112204 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" event={"ID":"73c500e2-aa65-4d5b-8571-6320da448d80","Type":"ContainerDied","Data":"3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2"} Jul 24 23:56:42.112373 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.112283 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h" event={"ID":"73c500e2-aa65-4d5b-8571-6320da448d80","Type":"ContainerDied","Data":"ba4773bf6b2e5b02e4b257509517aedaefbfa3183b12a4eaa95ac42a46b47b90"} Jul 24 23:56:42.112373 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.112308 2586 scope.go:117] "RemoveContainer" containerID="3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2" Jul 24 23:56:42.121313 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.121290 2586 scope.go:117] "RemoveContainer" containerID="3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2" Jul 24 23:56:42.121575 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:56:42.121555 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2\": container with ID starting with 3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2 not found: ID does not exist" containerID="3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2" Jul 24 23:56:42.121634 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.121583 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2"} err="failed to get container status \"3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2\": rpc error: code = NotFound desc = could not find container \"3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2\": container with ID starting with 3c8ad1ac36076aaf94256c4b490a38702145253e5b4707429ef08df1a7c7f6e2 not found: ID does not exist" Jul 24 23:56:42.132933 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.132901 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h"] Jul 24 23:56:42.136309 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.136280 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-90db14c6-77d446cd84-j6c7h"] Jul 24 23:56:42.787505 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:42.787471 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73c500e2-aa65-4d5b-8571-6320da448d80" path="/var/lib/kubelet/pods/73c500e2-aa65-4d5b-8571-6320da448d80/volumes" Jul 24 23:56:45.916333 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:45.916291 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5"] Jul 24 23:56:45.918229 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:45.918205 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-90db14c6-m7pp5"] Jul 24 23:56:46.059734 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:46.059696 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs"] Jul 24 23:56:46.060072 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:46.060022 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" containerID="cri-o://290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507" gracePeriod=30 Jul 24 23:56:46.786217 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:46.786174 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a98a9e6-797c-4548-96c8-181473e27225" path="/var/lib/kubelet/pods/9a98a9e6-797c-4548-96c8-181473e27225/volumes" Jul 24 23:56:47.140591 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.140559 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj"] Jul 24 23:56:47.141147 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.141127 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a98a9e6-797c-4548-96c8-181473e27225" containerName="revoke-keys" Jul 24 23:56:47.141232 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.141151 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a98a9e6-797c-4548-96c8-181473e27225" containerName="revoke-keys" Jul 24 23:56:47.141232 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.141185 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73c500e2-aa65-4d5b-8571-6320da448d80" containerName="payload-pre-processing" Jul 24 23:56:47.141232 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.141195 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="73c500e2-aa65-4d5b-8571-6320da448d80" containerName="payload-pre-processing" Jul 24 23:56:47.141481 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.141380 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="73c500e2-aa65-4d5b-8571-6320da448d80" containerName="payload-pre-processing" Jul 24 23:56:47.141481 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.141401 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a98a9e6-797c-4548-96c8-181473e27225" containerName="revoke-keys" Jul 24 23:56:47.146135 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.146114 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.148781 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.148760 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-25f28971-gw-openshift-default-dockercfg-9l6hd\"" Jul 24 23:56:47.170838 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.170809 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj"] Jul 24 23:56:47.288664 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288622 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-certs\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.288849 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288687 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-podinfo\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.288849 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288709 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-socket\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.288849 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288734 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-data\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.288849 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288759 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtn2j\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-kube-api-access-xtn2j\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.288849 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288807 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-envoy\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.289034 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288922 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-credential-socket\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.289034 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288950 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-token\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.289034 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.288974 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d91570-1f08-4637-b15c-b771b0e160c3-istiod-ca-cert\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390434 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-credential-socket\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390681 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390556 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-token\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390681 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390579 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d91570-1f08-4637-b15c-b771b0e160c3-istiod-ca-cert\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390681 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390632 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-certs\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390823 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390680 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-podinfo\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390823 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390705 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-socket\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.390823 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390740 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-data\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390842 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-credential-socket\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390889 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtn2j\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-kube-api-access-xtn2j\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.390929 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-envoy\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.391081 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-certs\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.391098 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-data\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.391175 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-socket\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.391346 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.391314 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d91570-1f08-4637-b15c-b771b0e160c3-istiod-ca-cert\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.393111 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.393088 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-envoy\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.393399 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.393378 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-podinfo\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.398666 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.398648 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtn2j\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-kube-api-access-xtn2j\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.398936 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.398914 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-token\") pod \"e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.457038 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.457001 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:47.606138 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.606109 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj"] Jul 24 23:56:47.608620 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:56:47.608592 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0d91570_1f08_4637_b15c_b771b0e160c3.slice/crio-d80f988bce72a31170973c8558d0f5033a8647479a4f8ab3dcc5e48b98572d34 WatchSource:0}: Error finding container d80f988bce72a31170973c8558d0f5033a8647479a4f8ab3dcc5e48b98572d34: Status 404 returned error can't find the container with id d80f988bce72a31170973c8558d0f5033a8647479a4f8ab3dcc5e48b98572d34 Jul 24 23:56:47.610802 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.610773 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:47.610875 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.610840 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:47.610918 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:47.610875 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:56:48.146894 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:48.146858 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" event={"ID":"a0d91570-1f08-4637-b15c-b771b0e160c3","Type":"ContainerStarted","Data":"d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb"} Jul 24 23:56:48.147284 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:48.146921 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" event={"ID":"a0d91570-1f08-4637-b15c-b771b0e160c3","Type":"ContainerStarted","Data":"d80f988bce72a31170973c8558d0f5033a8647479a4f8ab3dcc5e48b98572d34"} Jul 24 23:56:48.166739 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:48.166690 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" podStartSLOduration=1.166674942 podStartE2EDuration="1.166674942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:56:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:56:48.164755099 +0000 UTC m=+1667.857595928" watchObservedRunningTime="2026-07-24 23:56:48.166674942 +0000 UTC m=+1667.859515846" Jul 24 23:56:48.458156 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:48.458072 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:48.463043 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:48.463014 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:49.152223 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:49.152189 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:49.153055 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:49.153035 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:56:50.445505 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.445469 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc"] Jul 24 23:56:50.449851 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.449831 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.452144 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.452122 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-25f28971-dockercfg-ltv8r\"" Jul 24 23:56:50.452277 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.452169 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-25f28971\"" Jul 24 23:56:50.458136 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.458112 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc"] Jul 24 23:56:50.620167 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.620134 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052c1a6c-843d-44f8-a767-24b8cde4d1b1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.620167 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.620170 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjvnj\" (UniqueName: \"kubernetes.io/projected/052c1a6c-843d-44f8-a767-24b8cde4d1b1-kube-api-access-xjvnj\") pod \"payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.720999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.720920 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052c1a6c-843d-44f8-a767-24b8cde4d1b1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.720999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.720961 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjvnj\" (UniqueName: \"kubernetes.io/projected/052c1a6c-843d-44f8-a767-24b8cde4d1b1-kube-api-access-xjvnj\") pod \"payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.721608 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.721587 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052c1a6c-843d-44f8-a767-24b8cde4d1b1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.727895 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.727865 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjvnj\" (UniqueName: \"kubernetes.io/projected/052c1a6c-843d-44f8-a767-24b8cde4d1b1-kube-api-access-xjvnj\") pod \"payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:50.763181 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:50.763156 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:51.100824 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.100797 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc"] Jul 24 23:56:51.150163 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:56:51.150134 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052c1a6c_843d_44f8_a767_24b8cde4d1b1.slice/crio-59607640ef85c63c18bd756608652052521f0ee76e730380f1be7b292ca81b7c WatchSource:0}: Error finding container 59607640ef85c63c18bd756608652052521f0ee76e730380f1be7b292ca81b7c: Status 404 returned error can't find the container with id 59607640ef85c63c18bd756608652052521f0ee76e730380f1be7b292ca81b7c Jul 24 23:56:51.294036 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.294011 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:51.427558 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427530 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-certs\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.427747 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427569 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-data\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.427747 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427616 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-token\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.427747 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427658 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istiod-ca-cert\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.427747 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427698 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-podinfo\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.427747 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427728 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zq664\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-kube-api-access-zq664\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.428012 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427778 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-envoy\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.428012 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427820 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-socket\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.428012 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427824 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:56:51.428012 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.427847 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-credential-socket\") pod \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\" (UID: \"7cde3110-9aca-4b76-9ada-a9e75a7ab708\") " Jul 24 23:56:51.428282 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.428086 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:56:51.428282 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.428192 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.428282 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.428211 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.428447 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.428346 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:56:51.428447 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.428392 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:56:51.428557 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.428528 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:56:51.430008 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.429974 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:56:51.430127 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.430010 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:56:51.430127 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.430024 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-kube-api-access-zq664" (OuterVolumeSpecName: "kube-api-access-zq664") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "kube-api-access-zq664". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:56:51.430127 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.430075 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cde3110-9aca-4b76-9ada-a9e75a7ab708" (UID: "7cde3110-9aca-4b76-9ada-a9e75a7ab708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:56:51.529461 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529431 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.529461 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529457 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq664\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-kube-api-access-zq664\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.529461 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529467 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.529461 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529475 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.530270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529486 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.530270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529499 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:51.530270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:51.529506 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cde3110-9aca-4b76-9ada-a9e75a7ab708-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:56:52.169379 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.169344 2586 generic.go:356] "Generic (PLEG): container finished" podID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerID="290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507" exitCode=0 Jul 24 23:56:52.169531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.169389 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" Jul 24 23:56:52.169531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.169434 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" event={"ID":"7cde3110-9aca-4b76-9ada-a9e75a7ab708","Type":"ContainerDied","Data":"290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507"} Jul 24 23:56:52.169531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.169487 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs" event={"ID":"7cde3110-9aca-4b76-9ada-a9e75a7ab708","Type":"ContainerDied","Data":"34810ca460ae48ee165f73ef9175d4dbecc6e34d42621cb80dcb416a200146e1"} Jul 24 23:56:52.169531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.169513 2586 scope.go:117] "RemoveContainer" containerID="290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507" Jul 24 23:56:52.171131 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.171097 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" event={"ID":"052c1a6c-843d-44f8-a767-24b8cde4d1b1","Type":"ContainerStarted","Data":"c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7"} Jul 24 23:56:52.171270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.171149 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" event={"ID":"052c1a6c-843d-44f8-a767-24b8cde4d1b1","Type":"ContainerStarted","Data":"59607640ef85c63c18bd756608652052521f0ee76e730380f1be7b292ca81b7c"} Jul 24 23:56:52.171270 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.171184 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:56:52.179335 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.179316 2586 scope.go:117] "RemoveContainer" containerID="e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f" Jul 24 23:56:52.187307 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.187278 2586 scope.go:117] "RemoveContainer" containerID="290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507" Jul 24 23:56:52.187480 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.187440 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" podStartSLOduration=2.187425392 podStartE2EDuration="2.187425392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:56:52.186414089 +0000 UTC m=+1671.879254915" watchObservedRunningTime="2026-07-24 23:56:52.187425392 +0000 UTC m=+1671.880266230" Jul 24 23:56:52.187588 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:56:52.187567 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507\": container with ID starting with 290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507 not found: ID does not exist" containerID="290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507" Jul 24 23:56:52.187652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.187600 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507"} err="failed to get container status \"290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507\": rpc error: code = NotFound desc = could not find container \"290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507\": container with ID starting with 290b0aa7516b8ac98d25e5d5e634614d990f5cb0af1716aa691e3ef370dfe507 not found: ID does not exist" Jul 24 23:56:52.187652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.187625 2586 scope.go:117] "RemoveContainer" containerID="e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f" Jul 24 23:56:52.187882 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:56:52.187862 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f\": container with ID starting with e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f not found: ID does not exist" containerID="e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f" Jul 24 23:56:52.187922 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.187890 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f"} err="failed to get container status \"e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f\": rpc error: code = NotFound desc = could not find container \"e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f\": container with ID starting with e0fa21d14f596cfc154674aadd3bbea9f8ef8b48c72764641699602d28d7d70f not found: ID does not exist" Jul 24 23:56:52.199814 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.199794 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs"] Jul 24 23:56:52.205411 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.205391 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-90db14c6-openshift-default-54dbc56486-9mrfs"] Jul 24 23:56:52.787128 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:56:52.787100 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" path="/var/lib/kubelet/pods/7cde3110-9aca-4b76-9ada-a9e75a7ab708/volumes" Jul 24 23:57:01.224009 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:01.223913 2586 generic.go:356] "Generic (PLEG): container finished" podID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerID="d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb" exitCode=0 Jul 24 23:57:01.224009 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:01.223947 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" event={"ID":"a0d91570-1f08-4637-b15c-b771b0e160c3","Type":"ContainerDied","Data":"d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb"} Jul 24 23:57:01.224543 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:01.224335 2586 scope.go:117] "RemoveContainer" containerID="d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb" Jul 24 23:57:01.225365 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:01.225336 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:01.225479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:01.225402 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:01.225479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:01.225432 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:02.231165 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:02.231128 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" event={"ID":"a0d91570-1f08-4637-b15c-b771b0e160c3","Type":"ContainerStarted","Data":"ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830"} Jul 24 23:57:02.457651 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:02.457606 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:57:02.458593 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:02.458567 2586 patch_prober.go:28] interesting pod/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 24 23:57:02.458801 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:02.458775 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 24 23:57:03.179988 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:03.179959 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:57:03.458256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:03.458136 2586 patch_prober.go:28] interesting pod/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 24 23:57:03.458256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:03.458218 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 24 23:57:04.152967 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:04.152931 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:57:04.458053 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:04.457957 2586 patch_prober.go:28] interesting pod/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 24 23:57:04.458053 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:04.458022 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 24 23:57:05.457965 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:05.457927 2586 patch_prober.go:28] interesting pod/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Jul 24 23:57:05.458553 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:05.458522 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Jul 24 23:57:06.461256 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:06.461215 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:57:06.462283 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:06.462261 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:57:25.250129 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.250041 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc"] Jul 24 23:57:25.250709 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.250277 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" podUID="052c1a6c-843d-44f8-a767-24b8cde4d1b1" containerName="payload-processing" containerID="cri-o://c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7" gracePeriod=30 Jul 24 23:57:25.507674 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.507609 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:57:25.642151 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.642116 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjvnj\" (UniqueName: \"kubernetes.io/projected/052c1a6c-843d-44f8-a767-24b8cde4d1b1-kube-api-access-xjvnj\") pod \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " Jul 24 23:57:25.642356 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.642218 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052c1a6c-843d-44f8-a767-24b8cde4d1b1-plugins-config-volume\") pod \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\" (UID: \"052c1a6c-843d-44f8-a767-24b8cde4d1b1\") " Jul 24 23:57:25.642595 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.642569 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052c1a6c-843d-44f8-a767-24b8cde4d1b1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "052c1a6c-843d-44f8-a767-24b8cde4d1b1" (UID: "052c1a6c-843d-44f8-a767-24b8cde4d1b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:57:25.644279 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.644250 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052c1a6c-843d-44f8-a767-24b8cde4d1b1-kube-api-access-xjvnj" (OuterVolumeSpecName: "kube-api-access-xjvnj") pod "052c1a6c-843d-44f8-a767-24b8cde4d1b1" (UID: "052c1a6c-843d-44f8-a767-24b8cde4d1b1"). InnerVolumeSpecName "kube-api-access-xjvnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:57:25.743819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.743788 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjvnj\" (UniqueName: \"kubernetes.io/projected/052c1a6c-843d-44f8-a767-24b8cde4d1b1-kube-api-access-xjvnj\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:25.743819 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:25.743816 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/052c1a6c-843d-44f8-a767-24b8cde4d1b1-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:26.364896 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.364853 2586 generic.go:356] "Generic (PLEG): container finished" podID="052c1a6c-843d-44f8-a767-24b8cde4d1b1" containerID="c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7" exitCode=0 Jul 24 23:57:26.365370 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.364918 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" Jul 24 23:57:26.365370 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.364942 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" event={"ID":"052c1a6c-843d-44f8-a767-24b8cde4d1b1","Type":"ContainerDied","Data":"c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7"} Jul 24 23:57:26.365370 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.365021 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc" event={"ID":"052c1a6c-843d-44f8-a767-24b8cde4d1b1","Type":"ContainerDied","Data":"59607640ef85c63c18bd756608652052521f0ee76e730380f1be7b292ca81b7c"} Jul 24 23:57:26.365370 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.365047 2586 scope.go:117] "RemoveContainer" containerID="c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7" Jul 24 23:57:26.374501 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.374485 2586 scope.go:117] "RemoveContainer" containerID="c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7" Jul 24 23:57:26.374750 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:57:26.374732 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7\": container with ID starting with c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7 not found: ID does not exist" containerID="c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7" Jul 24 23:57:26.374797 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.374758 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7"} err="failed to get container status \"c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7\": rpc error: code = NotFound desc = could not find container \"c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7\": container with ID starting with c9b170c2dc2b5d808b382d752397fa7d6c4dbd7ee7eb90dd57a23346b81344f7 not found: ID does not exist" Jul 24 23:57:26.385496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.385471 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc"] Jul 24 23:57:26.390857 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.390836 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-25f28971-557d48bfb9-2w7cc"] Jul 24 23:57:26.786890 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:26.786806 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052c1a6c-843d-44f8-a767-24b8cde4d1b1" path="/var/lib/kubelet/pods/052c1a6c-843d-44f8-a767-24b8cde4d1b1/volumes" Jul 24 23:57:29.878206 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:29.878166 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj"] Jul 24 23:57:29.878723 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:29.878437 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" containerID="cri-o://ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830" gracePeriod=30 Jul 24 23:57:30.714136 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714095 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw"] Jul 24 23:57:30.714736 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714715 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" Jul 24 23:57:30.714873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714740 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" Jul 24 23:57:30.714873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714778 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052c1a6c-843d-44f8-a767-24b8cde4d1b1" containerName="payload-processing" Jul 24 23:57:30.714873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714787 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="052c1a6c-843d-44f8-a767-24b8cde4d1b1" containerName="payload-processing" Jul 24 23:57:30.714873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714859 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="052c1a6c-843d-44f8-a767-24b8cde4d1b1" containerName="payload-processing" Jul 24 23:57:30.714873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714869 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" Jul 24 23:57:30.714873 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.714875 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" Jul 24 23:57:30.719637 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.719615 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.721837 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.721817 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0c7d785d-openshift-default-dockercfg-5hjzz\"" Jul 24 23:57:30.727215 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.727191 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw"] Jul 24 23:57:30.794456 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794422 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-data\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794464 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-token\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794711 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794659 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-socket\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794764 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794707 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-certs\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794764 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794748 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-credential-socket\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794857 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794765 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-podinfo\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794857 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794790 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v25d\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-kube-api-access-7v25d\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794855 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-envoy\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.794954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.794880 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istiod-ca-cert\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896052 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.895991 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-token\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896094 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-socket\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896127 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-certs\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896194 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-credential-socket\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896221 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-podinfo\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896286 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v25d\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-kube-api-access-7v25d\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896350 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-envoy\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896378 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istiod-ca-cert\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.896625 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-data\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.897008 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896614 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-socket\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.897008 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.896781 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-certs\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.897115 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.897053 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-data\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.897115 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.897060 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-credential-socket\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.897506 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.897473 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istiod-ca-cert\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.898890 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.898873 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-envoy\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.899021 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.899003 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-podinfo\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.903500 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.903478 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-token\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:30.903601 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:30.903554 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v25d\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-kube-api-access-7v25d\") pod \"e2e-ait-0c7d785d-openshift-default-576999f564-nktbw\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:31.033141 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.033058 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:31.174656 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.174629 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw"] Jul 24 23:57:31.176869 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:57:31.176842 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e1e137_cb22_4ae0_9996_e8d051f4915d.slice/crio-9777198eaae26d9a5c177af402dfcf61c97900bd992aacbb9b8633b8f42021f3 WatchSource:0}: Error finding container 9777198eaae26d9a5c177af402dfcf61c97900bd992aacbb9b8633b8f42021f3: Status 404 returned error can't find the container with id 9777198eaae26d9a5c177af402dfcf61c97900bd992aacbb9b8633b8f42021f3 Jul 24 23:57:31.179448 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.179417 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:31.179538 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.179484 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:31.179538 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.179513 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:31.393913 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.393877 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" event={"ID":"c6e1e137-cb22-4ae0-9996-e8d051f4915d","Type":"ContainerStarted","Data":"19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43"} Jul 24 23:57:31.394110 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.393939 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" event={"ID":"c6e1e137-cb22-4ae0-9996-e8d051f4915d","Type":"ContainerStarted","Data":"9777198eaae26d9a5c177af402dfcf61c97900bd992aacbb9b8633b8f42021f3"} Jul 24 23:57:31.412701 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:31.412649 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" podStartSLOduration=1.41263312 podStartE2EDuration="1.41263312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:57:31.410004744 +0000 UTC m=+1711.102845570" watchObservedRunningTime="2026-07-24 23:57:31.41263312 +0000 UTC m=+1711.105473947" Jul 24 23:57:32.033561 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:32.033528 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:32.039134 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:32.039105 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:32.403095 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:32.403061 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:32.404142 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:32.404118 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:33.641662 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.641630 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw"] Jul 24 23:57:33.642160 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.642140 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" Jul 24 23:57:33.642160 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.642160 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cde3110-9aca-4b76-9ada-a9e75a7ab708" containerName="istio-proxy" Jul 24 23:57:33.645122 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.645090 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.647560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.647541 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0c7d785d-dockercfg-lrqlg\"" Jul 24 23:57:33.647668 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.647566 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0c7d785d\"" Jul 24 23:57:33.654704 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.654678 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw"] Jul 24 23:57:33.723014 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.722978 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm7l5\" (UniqueName: \"kubernetes.io/projected/b4907492-8aa3-4867-ab96-4966e608de21-kube-api-access-mm7l5\") pod \"payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.723250 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.723215 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4907492-8aa3-4867-ab96-4966e608de21-plugins-config-volume\") pod \"payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.824550 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.824517 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4907492-8aa3-4867-ab96-4966e608de21-plugins-config-volume\") pod \"payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.824722 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.824580 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm7l5\" (UniqueName: \"kubernetes.io/projected/b4907492-8aa3-4867-ab96-4966e608de21-kube-api-access-mm7l5\") pod \"payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.825212 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.825193 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4907492-8aa3-4867-ab96-4966e608de21-plugins-config-volume\") pod \"payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.831857 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.831835 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm7l5\" (UniqueName: \"kubernetes.io/projected/b4907492-8aa3-4867-ab96-4966e608de21-kube-api-access-mm7l5\") pod \"payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:33.958135 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:33.958054 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:34.090347 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:34.090323 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw"] Jul 24 23:57:34.092036 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:57:34.091992 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4907492_8aa3_4867_ab96_4966e608de21.slice/crio-2cd8aefd0624be81f703e7ba286c5c90cb9804c77282dfa6a007ebb8abb89447 WatchSource:0}: Error finding container 2cd8aefd0624be81f703e7ba286c5c90cb9804c77282dfa6a007ebb8abb89447: Status 404 returned error can't find the container with id 2cd8aefd0624be81f703e7ba286c5c90cb9804c77282dfa6a007ebb8abb89447 Jul 24 23:57:34.414627 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:34.414594 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" event={"ID":"b4907492-8aa3-4867-ab96-4966e608de21","Type":"ContainerStarted","Data":"f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb"} Jul 24 23:57:34.414863 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:34.414845 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:34.414909 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:34.414886 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" event={"ID":"b4907492-8aa3-4867-ab96-4966e608de21","Type":"ContainerStarted","Data":"2cd8aefd0624be81f703e7ba286c5c90cb9804c77282dfa6a007ebb8abb89447"} Jul 24 23:57:34.430620 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:34.430577 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" podStartSLOduration=1.430559815 podStartE2EDuration="1.430559815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:57:34.429657871 +0000 UTC m=+1714.122498697" watchObservedRunningTime="2026-07-24 23:57:34.430559815 +0000 UTC m=+1714.123400643" Jul 24 23:57:35.127794 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.127768 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:57:35.236171 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236086 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-certs\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236171 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236118 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-envoy\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236171 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236155 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-token\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236480 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236231 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-data\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236480 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236382 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d91570-1f08-4637-b15c-b771b0e160c3-istiod-ca-cert\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236480 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236451 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-credential-socket\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236637 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236496 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-podinfo\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236637 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236389 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:57:35.236637 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236522 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-socket\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.236795 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236737 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:57:35.236795 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236780 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0d91570-1f08-4637-b15c-b771b0e160c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:57:35.236904 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236786 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:57:35.236904 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.236813 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xtn2j\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-kube-api-access-xtn2j\") pod \"a0d91570-1f08-4637-b15c-b771b0e160c3\" (UID: \"a0d91570-1f08-4637-b15c-b771b0e160c3\") " Jul 24 23:57:35.238640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.237143 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0d91570-1f08-4637-b15c-b771b0e160c3-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.238640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.237166 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.238640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.237181 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.238640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.237194 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.238640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.237301 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:57:35.238640 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.238555 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:57:35.239015 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.238891 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:57:35.239143 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.239123 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-kube-api-access-xtn2j" (OuterVolumeSpecName: "kube-api-access-xtn2j") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "kube-api-access-xtn2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:57:35.239224 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.239168 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a0d91570-1f08-4637-b15c-b771b0e160c3" (UID: "a0d91570-1f08-4637-b15c-b771b0e160c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:57:35.338074 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.338052 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.338074 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.338072 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.338193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.338080 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.338193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.338088 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtn2j\" (UniqueName: \"kubernetes.io/projected/a0d91570-1f08-4637-b15c-b771b0e160c3-kube-api-access-xtn2j\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.338193 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.338097 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0d91570-1f08-4637-b15c-b771b0e160c3-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:57:35.421127 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.421104 2586 generic.go:356] "Generic (PLEG): container finished" podID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerID="ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830" exitCode=0 Jul 24 23:57:35.421274 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.421151 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" Jul 24 23:57:35.421274 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.421198 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" event={"ID":"a0d91570-1f08-4637-b15c-b771b0e160c3","Type":"ContainerDied","Data":"ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830"} Jul 24 23:57:35.421404 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.421293 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj" event={"ID":"a0d91570-1f08-4637-b15c-b771b0e160c3","Type":"ContainerDied","Data":"d80f988bce72a31170973c8558d0f5033a8647479a4f8ab3dcc5e48b98572d34"} Jul 24 23:57:35.421404 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.421330 2586 scope.go:117] "RemoveContainer" containerID="ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830" Jul 24 23:57:35.430420 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.430404 2586 scope.go:117] "RemoveContainer" containerID="d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb" Jul 24 23:57:35.439726 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.439707 2586 scope.go:117] "RemoveContainer" containerID="ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830" Jul 24 23:57:35.440268 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:57:35.440231 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830\": container with ID starting with ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830 not found: ID does not exist" containerID="ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830" Jul 24 23:57:35.440366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.440278 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830"} err="failed to get container status \"ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830\": rpc error: code = NotFound desc = could not find container \"ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830\": container with ID starting with ba3b61844ae297d46784284beb3910b97f55df1f1a434d2ff82986f03b436830 not found: ID does not exist" Jul 24 23:57:35.440366 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.440301 2586 scope.go:117] "RemoveContainer" containerID="d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb" Jul 24 23:57:35.440584 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:57:35.440567 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb\": container with ID starting with d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb not found: ID does not exist" containerID="d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb" Jul 24 23:57:35.440652 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.440592 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb"} err="failed to get container status \"d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb\": rpc error: code = NotFound desc = could not find container \"d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb\": container with ID starting with d8b12d936ae37c4d5b2f96bc8324a3c7641e07d1c9f4e7e0c8094b5890cde3cb not found: ID does not exist" Jul 24 23:57:35.441566 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.441548 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj"] Jul 24 23:57:35.445911 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:35.445890 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-25f28971-gw-openshift-default-5d995f8b8d-xqptj"] Jul 24 23:57:36.786803 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:36.786770 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" path="/var/lib/kubelet/pods/a0d91570-1f08-4637-b15c-b771b0e160c3/volumes" Jul 24 23:57:45.424503 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:45.424471 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:57:46.492780 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:46.492742 2586 generic.go:356] "Generic (PLEG): container finished" podID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerID="19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43" exitCode=0 Jul 24 23:57:46.493227 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:46.492818 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" event={"ID":"c6e1e137-cb22-4ae0-9996-e8d051f4915d","Type":"ContainerDied","Data":"19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43"} Jul 24 23:57:46.493297 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:46.493272 2586 scope.go:117] "RemoveContainer" containerID="19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43" Jul 24 23:57:46.494266 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:46.494218 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:46.494324 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:46.494310 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:46.494363 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:46.494343 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:57:47.403111 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:47.403075 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:47.499858 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:47.499822 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" event={"ID":"c6e1e137-cb22-4ae0-9996-e8d051f4915d","Type":"ContainerStarted","Data":"67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c"} Jul 24 23:57:48.033471 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:48.033438 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:48.034447 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:48.034413 2586 patch_prober.go:28] interesting pod/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 24 23:57:48.034556 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:48.034475 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 24 23:57:49.034121 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:49.034085 2586 patch_prober.go:28] interesting pod/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 24 23:57:49.034655 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:49.034139 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 24 23:57:50.033452 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:50.033419 2586 patch_prober.go:28] interesting pod/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 24 23:57:50.033634 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:50.033475 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 24 23:57:51.034034 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:51.034002 2586 patch_prober.go:28] interesting pod/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 24 23:57:51.034495 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:51.034062 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 24 23:57:52.037497 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:52.037468 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:52.037872 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:52.037798 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:57:52.038424 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:57:52.038408 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:58:09.247316 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.247282 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw"] Jul 24 23:58:09.247850 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.247501 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" podUID="b4907492-8aa3-4867-ab96-4966e608de21" containerName="payload-processing" containerID="cri-o://f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb" gracePeriod=30 Jul 24 23:58:09.507954 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.507893 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:58:09.613692 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.613660 2586 generic.go:356] "Generic (PLEG): container finished" podID="b4907492-8aa3-4867-ab96-4966e608de21" containerID="f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb" exitCode=0 Jul 24 23:58:09.613862 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.613707 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" Jul 24 23:58:09.613862 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.613751 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" event={"ID":"b4907492-8aa3-4867-ab96-4966e608de21","Type":"ContainerDied","Data":"f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb"} Jul 24 23:58:09.613862 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.613809 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw" event={"ID":"b4907492-8aa3-4867-ab96-4966e608de21","Type":"ContainerDied","Data":"2cd8aefd0624be81f703e7ba286c5c90cb9804c77282dfa6a007ebb8abb89447"} Jul 24 23:58:09.613862 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.613839 2586 scope.go:117] "RemoveContainer" containerID="f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb" Jul 24 23:58:09.622282 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.622264 2586 scope.go:117] "RemoveContainer" containerID="f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb" Jul 24 23:58:09.622516 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:58:09.622499 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb\": container with ID starting with f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb not found: ID does not exist" containerID="f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb" Jul 24 23:58:09.622568 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.622525 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb"} err="failed to get container status \"f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb\": rpc error: code = NotFound desc = could not find container \"f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb\": container with ID starting with f74463a8686f0556172d5ebf9b568ad0ac2cf480bdaead816232a6e62e67f9cb not found: ID does not exist" Jul 24 23:58:09.629828 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.629809 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mm7l5\" (UniqueName: \"kubernetes.io/projected/b4907492-8aa3-4867-ab96-4966e608de21-kube-api-access-mm7l5\") pod \"b4907492-8aa3-4867-ab96-4966e608de21\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " Jul 24 23:58:09.629989 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.629977 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4907492-8aa3-4867-ab96-4966e608de21-plugins-config-volume\") pod \"b4907492-8aa3-4867-ab96-4966e608de21\" (UID: \"b4907492-8aa3-4867-ab96-4966e608de21\") " Jul 24 23:58:09.630348 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.630322 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4907492-8aa3-4867-ab96-4966e608de21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b4907492-8aa3-4867-ab96-4966e608de21" (UID: "b4907492-8aa3-4867-ab96-4966e608de21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:58:09.631799 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.631774 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4907492-8aa3-4867-ab96-4966e608de21-kube-api-access-mm7l5" (OuterVolumeSpecName: "kube-api-access-mm7l5") pod "b4907492-8aa3-4867-ab96-4966e608de21" (UID: "b4907492-8aa3-4867-ab96-4966e608de21"). InnerVolumeSpecName "kube-api-access-mm7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:58:09.731145 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.731110 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm7l5\" (UniqueName: \"kubernetes.io/projected/b4907492-8aa3-4867-ab96-4966e608de21-kube-api-access-mm7l5\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:09.731145 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.731140 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4907492-8aa3-4867-ab96-4966e608de21-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:09.934836 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.934802 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw"] Jul 24 23:58:09.943711 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:09.943683 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0c7d785d-78c7877d56-k8bpw"] Jul 24 23:58:10.786204 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:10.786170 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4907492-8aa3-4867-ab96-4966e608de21" path="/var/lib/kubelet/pods/b4907492-8aa3-4867-ab96-4966e608de21/volumes" Jul 24 23:58:14.994847 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:14.994808 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw"] Jul 24 23:58:14.995318 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:14.995093 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" containerID="cri-o://67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c" gracePeriod=30 Jul 24 23:58:15.775806 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.775765 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl"] Jul 24 23:58:15.776417 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776399 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4907492-8aa3-4867-ab96-4966e608de21" containerName="payload-processing" Jul 24 23:58:15.776496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776420 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4907492-8aa3-4867-ab96-4966e608de21" containerName="payload-processing" Jul 24 23:58:15.776496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776434 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" Jul 24 23:58:15.776496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776444 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" Jul 24 23:58:15.776496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776455 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" Jul 24 23:58:15.776496 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776463 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" Jul 24 23:58:15.776672 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776563 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4907492-8aa3-4867-ab96-4966e608de21" containerName="payload-processing" Jul 24 23:58:15.776672 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776574 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" Jul 24 23:58:15.776672 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.776583 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d91570-1f08-4637-b15c-b771b0e160c3" containerName="istio-proxy" Jul 24 23:58:15.781133 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.781108 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.783629 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.783605 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-9ffa79f3-openshift-default-dockercfg-9lht2\"" Jul 24 23:58:15.790599 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.790573 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl"] Jul 24 23:58:15.883106 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883061 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-credential-socket\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883321 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883124 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-socket\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883321 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883152 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-token\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883321 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883183 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pth5f\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-kube-api-access-pth5f\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883321 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883213 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-envoy\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883321 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883326 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-podinfo\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883687 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883440 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-certs\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883687 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883515 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad876a6-05a6-4977-b658-3752521a5d7c-istiod-ca-cert\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.883687 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.883581 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-data\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984510 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984479 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-socket\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984510 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984512 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-token\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984676 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984529 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pth5f\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-kube-api-access-pth5f\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984676 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984638 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-envoy\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984774 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984687 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-podinfo\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984774 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984725 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-certs\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984774 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984748 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad876a6-05a6-4977-b658-3752521a5d7c-istiod-ca-cert\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984927 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984787 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-data\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.984927 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984881 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-credential-socket\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.985029 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.984929 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-socket\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.985168 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.985144 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-certs\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.985230 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.985206 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-data\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.985230 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.985220 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-credential-socket\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.985477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.985454 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad876a6-05a6-4977-b658-3752521a5d7c-istiod-ca-cert\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.987014 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.986985 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-envoy\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.987180 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.987113 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-podinfo\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.992675 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.992655 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-token\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:15.992879 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:15.992862 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pth5f\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-kube-api-access-pth5f\") pod \"e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:16.093340 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.093267 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:16.232362 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.232336 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl"] Jul 24 23:58:16.239100 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:58:16.239069 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ad876a6_05a6_4977_b658_3752521a5d7c.slice/crio-be6c93a93cff5059b37ab1ce81d39fcfb79b9b85c59595e25b26a6c0b6962ea7 WatchSource:0}: Error finding container be6c93a93cff5059b37ab1ce81d39fcfb79b9b85c59595e25b26a6c0b6962ea7: Status 404 returned error can't find the container with id be6c93a93cff5059b37ab1ce81d39fcfb79b9b85c59595e25b26a6c0b6962ea7 Jul 24 23:58:16.241300 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.241160 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:58:16.241476 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.241446 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:58:16.241560 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.241516 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:58:16.650836 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.650797 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" event={"ID":"3ad876a6-05a6-4977-b658-3752521a5d7c","Type":"ContainerStarted","Data":"3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093"} Jul 24 23:58:16.651013 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.650848 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" event={"ID":"3ad876a6-05a6-4977-b658-3752521a5d7c","Type":"ContainerStarted","Data":"be6c93a93cff5059b37ab1ce81d39fcfb79b9b85c59595e25b26a6c0b6962ea7"} Jul 24 23:58:16.672305 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:16.672222 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podStartSLOduration=1.6722032549999999 podStartE2EDuration="1.672203255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:58:16.669381084 +0000 UTC m=+1756.362221909" watchObservedRunningTime="2026-07-24 23:58:16.672203255 +0000 UTC m=+1756.365044085" Jul 24 23:58:17.094395 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:17.094309 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:17.098875 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:17.098850 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:17.656033 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:17.655999 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:17.657100 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:17.657081 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:20.275934 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.275869 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:58:20.422844 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422815 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-data\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.422844 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422848 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-certs\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422893 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istiod-ca-cert\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422921 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-token\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422936 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7v25d\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-kube-api-access-7v25d\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422962 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-socket\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.422994 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-podinfo\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423027 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-envoy\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423075 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423077 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-credential-socket\") pod \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\" (UID: \"c6e1e137-cb22-4ae0-9996-e8d051f4915d\") " Jul 24 23:58:20.423459 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423144 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:58:20.423459 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423381 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:58:20.423563 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423457 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.423563 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423474 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:58:20.423758 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423661 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:58:20.423812 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.423790 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-data" (OuterVolumeSpecName: "istio-data") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:58:20.425330 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.425299 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:58:20.425430 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.425365 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:58:20.425568 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.425545 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-token" (OuterVolumeSpecName: "istio-token") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:58:20.425684 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.425650 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-kube-api-access-7v25d" (OuterVolumeSpecName: "kube-api-access-7v25d") pod "c6e1e137-cb22-4ae0-9996-e8d051f4915d" (UID: "c6e1e137-cb22-4ae0-9996-e8d051f4915d"). InnerVolumeSpecName "kube-api-access-7v25d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:58:20.524107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524080 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524107 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524105 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524118 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524132 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524142 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524152 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v25d\" (UniqueName: \"kubernetes.io/projected/c6e1e137-cb22-4ae0-9996-e8d051f4915d-kube-api-access-7v25d\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524165 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6e1e137-cb22-4ae0-9996-e8d051f4915d-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.524304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.524177 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6e1e137-cb22-4ae0-9996-e8d051f4915d-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:20.672044 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.672017 2586 generic.go:356] "Generic (PLEG): container finished" podID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerID="67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c" exitCode=0 Jul 24 23:58:20.672194 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.672066 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" Jul 24 23:58:20.672194 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.672103 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" event={"ID":"c6e1e137-cb22-4ae0-9996-e8d051f4915d","Type":"ContainerDied","Data":"67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c"} Jul 24 23:58:20.672194 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.672162 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw" event={"ID":"c6e1e137-cb22-4ae0-9996-e8d051f4915d","Type":"ContainerDied","Data":"9777198eaae26d9a5c177af402dfcf61c97900bd992aacbb9b8633b8f42021f3"} Jul 24 23:58:20.672405 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.672194 2586 scope.go:117] "RemoveContainer" containerID="67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c" Jul 24 23:58:20.681934 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.681917 2586 scope.go:117] "RemoveContainer" containerID="19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43" Jul 24 23:58:20.689531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.689512 2586 scope.go:117] "RemoveContainer" containerID="67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c" Jul 24 23:58:20.689785 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:58:20.689767 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c\": container with ID starting with 67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c not found: ID does not exist" containerID="67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c" Jul 24 23:58:20.689829 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.689792 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c"} err="failed to get container status \"67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c\": rpc error: code = NotFound desc = could not find container \"67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c\": container with ID starting with 67a7338661c64171abfd4c0c08efbdbf8dde4ffe91c20c591089d774696dd37c not found: ID does not exist" Jul 24 23:58:20.689829 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.689809 2586 scope.go:117] "RemoveContainer" containerID="19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43" Jul 24 23:58:20.690037 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:58:20.690016 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43\": container with ID starting with 19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43 not found: ID does not exist" containerID="19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43" Jul 24 23:58:20.690087 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.690043 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43"} err="failed to get container status \"19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43\": rpc error: code = NotFound desc = could not find container \"19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43\": container with ID starting with 19f8bbd8206cf9f42202525fdca4d8b0b364d37261470abe82b7388f5cb67a43 not found: ID does not exist" Jul 24 23:58:20.692857 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.692839 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw"] Jul 24 23:58:20.698468 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.698444 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0c7d785d-openshift-default-576999f564-nktbw"] Jul 24 23:58:20.786546 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:20.786515 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" path="/var/lib/kubelet/pods/c6e1e137-cb22-4ae0-9996-e8d051f4915d/volumes" Jul 24 23:58:32.805624 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:32.805576 2586 patch_prober.go:28] interesting pod/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:46058->10.132.0.56:15021: read: connection reset by peer" start-of-body= Jul 24 23:58:32.805977 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:32.805669 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": read tcp 10.132.0.2:46058->10.132.0.56:15021: read: connection reset by peer" Jul 24 23:58:33.744074 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:33.744036 2586 generic.go:356] "Generic (PLEG): container finished" podID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerID="3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093" exitCode=0 Jul 24 23:58:33.744314 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:33.744116 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" event={"ID":"3ad876a6-05a6-4977-b658-3752521a5d7c","Type":"ContainerDied","Data":"3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093"} Jul 24 23:58:33.744606 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:33.744589 2586 scope.go:117] "RemoveContainer" containerID="3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093" Jul 24 23:58:33.745859 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:33.745830 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:58:33.745938 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:33.745920 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:58:33.745975 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:33.745964 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 24 23:58:34.757287 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:34.757215 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" event={"ID":"3ad876a6-05a6-4977-b658-3752521a5d7c","Type":"ContainerStarted","Data":"2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6"} Jul 24 23:58:35.094305 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:35.094169 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:35.094479 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:35.094455 2586 patch_prober.go:28] interesting pod/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 23:58:35.094540 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:35.094520 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 23:58:36.094531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:36.094497 2586 patch_prober.go:28] interesting pod/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 23:58:36.094914 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:36.094564 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 23:58:37.094584 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:37.094546 2586 patch_prober.go:28] interesting pod/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 23:58:37.094983 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:37.094604 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 23:58:38.094567 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:38.094533 2586 patch_prober.go:28] interesting pod/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 23:58:38.094757 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:38.094591 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 23:58:39.097820 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:39.097790 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:39.098289 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:39.098193 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:39.098848 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:39.098828 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:58:42.925520 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925489 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v"] Jul 24 23:58:42.925905 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925862 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" Jul 24 23:58:42.925905 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925872 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" Jul 24 23:58:42.925905 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925881 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" Jul 24 23:58:42.925905 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925887 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" Jul 24 23:58:42.926037 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925960 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" Jul 24 23:58:42.926037 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.925969 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6e1e137-cb22-4ae0-9996-e8d051f4915d" containerName="istio-proxy" Jul 24 23:58:42.929025 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.929008 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:42.931316 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.931294 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 23:58:42.931440 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.931400 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bxr46\"" Jul 24 23:58:42.935679 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:42.935632 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v"] Jul 24 23:58:43.034043 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.034018 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v\" (UID: \"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:43.134986 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.134962 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v\" (UID: \"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:43.135569 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.135544 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v\" (UID: \"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:43.241641 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.241582 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:43.373465 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.373433 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v"] Jul 24 23:58:43.374998 ip-10-0-131-63 kubenswrapper[2586]: W0724 23:58:43.374971 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d880556_4bdc_4fb4_86d4_ab0a231ac3d1.slice/crio-9d6d67de4b96382766c1e6aff2958a1eddb8e6197606316f5865ff69c7fc4851 WatchSource:0}: Error finding container 9d6d67de4b96382766c1e6aff2958a1eddb8e6197606316f5865ff69c7fc4851: Status 404 returned error can't find the container with id 9d6d67de4b96382766c1e6aff2958a1eddb8e6197606316f5865ff69c7fc4851 Jul 24 23:58:43.805841 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.805766 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" event={"ID":"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1","Type":"ContainerStarted","Data":"42f6fbca2c7b10e0c1771c50e69c9eb90c9e97b2be895d2e88ac83f29e8b990a"} Jul 24 23:58:43.805841 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.805824 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" event={"ID":"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1","Type":"ContainerStarted","Data":"9d6d67de4b96382766c1e6aff2958a1eddb8e6197606316f5865ff69c7fc4851"} Jul 24 23:58:43.821187 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:43.821138 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" podStartSLOduration=1.821123729 podStartE2EDuration="1.821123729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 23:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 23:58:43.819431783 +0000 UTC m=+1783.512272610" watchObservedRunningTime="2026-07-24 23:58:43.821123729 +0000 UTC m=+1783.513964557" Jul 24 23:58:44.811837 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:44.811803 2586 generic.go:356] "Generic (PLEG): container finished" podID="2d880556-4bdc-4fb4-86d4-ab0a231ac3d1" containerID="42f6fbca2c7b10e0c1771c50e69c9eb90c9e97b2be895d2e88ac83f29e8b990a" exitCode=0 Jul 24 23:58:44.812277 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:44.811885 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" event={"ID":"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1","Type":"ContainerDied","Data":"42f6fbca2c7b10e0c1771c50e69c9eb90c9e97b2be895d2e88ac83f29e8b990a"} Jul 24 23:58:45.950208 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:45.950183 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:46.058933 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:46.058886 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1-maas-api-service-ca\") pod \"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1\" (UID: \"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1\") " Jul 24 23:58:46.059327 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:46.059302 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d880556-4bdc-4fb4-86d4-ab0a231ac3d1" (UID: "2d880556-4bdc-4fb4-86d4-ab0a231ac3d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:58:46.160310 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:46.160278 2586 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1-maas-api-service-ca\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:58:46.823442 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:46.823410 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" Jul 24 23:58:46.823442 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:46.823422 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v" event={"ID":"2d880556-4bdc-4fb4-86d4-ab0a231ac3d1","Type":"ContainerDied","Data":"9d6d67de4b96382766c1e6aff2958a1eddb8e6197606316f5865ff69c7fc4851"} Jul 24 23:58:46.823644 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:46.823469 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d6d67de4b96382766c1e6aff2958a1eddb8e6197606316f5865ff69c7fc4851" Jul 24 23:58:58.002382 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:58.002296 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v"] Jul 24 23:58:58.005494 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:58.005467 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9ffa79f3-9k52v"] Jul 24 23:58:58.146920 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:58.146886 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl"] Jul 24 23:58:58.147157 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:58.147130 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" containerID="cri-o://2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6" gracePeriod=30 Jul 24 23:58:58.787055 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:58:58.787012 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d880556-4bdc-4fb4-86d4-ab0a231ac3d1" path="/var/lib/kubelet/pods/2d880556-4bdc-4fb4-86d4-ab0a231ac3d1/volumes" Jul 24 23:59:00.886917 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:00.886889 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:59:00.889995 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:00.889971 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 24 23:59:03.401304 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.401276 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:59:03.511025 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.510940 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad876a6-05a6-4977-b658-3752521a5d7c-istiod-ca-cert\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511025 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.510985 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-podinfo\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511025 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511025 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-certs\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511054 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-credential-socket\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511099 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-socket\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511136 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pth5f\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-kube-api-access-pth5f\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511165 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-data\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511189 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-token\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511299 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-envoy\") pod \"3ad876a6-05a6-4977-b658-3752521a5d7c\" (UID: \"3ad876a6-05a6-4977-b658-3752521a5d7c\") " Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511381 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511399 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:59:03.511477 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511435 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ad876a6-05a6-4977-b658-3752521a5d7c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 23:59:03.511907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511627 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ad876a6-05a6-4977-b658-3752521a5d7c-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.511907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511640 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.511907 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511648 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.512065 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.511955 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:59:03.512279 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.512221 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-data" (OuterVolumeSpecName: "istio-data") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:59:03.513531 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.513501 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 23:59:03.513935 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.513916 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-kube-api-access-pth5f" (OuterVolumeSpecName: "kube-api-access-pth5f") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "kube-api-access-pth5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:59:03.513999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.513936 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 23:59:03.513999 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.513933 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-token" (OuterVolumeSpecName: "istio-token") pod "3ad876a6-05a6-4977-b658-3752521a5d7c" (UID: "3ad876a6-05a6-4977-b658-3752521a5d7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 23:59:03.612190 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.612162 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.612190 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.612186 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.612190 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.612195 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pth5f\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-kube-api-access-pth5f\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.612406 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.612204 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.612406 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.612213 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.612406 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.612220 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ad876a6-05a6-4977-b658-3752521a5d7c-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 24 23:59:03.913917 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.913884 2586 generic.go:356] "Generic (PLEG): container finished" podID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerID="2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6" exitCode=0 Jul 24 23:59:03.914098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.913955 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" Jul 24 23:59:03.914098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.913976 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" event={"ID":"3ad876a6-05a6-4977-b658-3752521a5d7c","Type":"ContainerDied","Data":"2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6"} Jul 24 23:59:03.914098 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.914074 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl" event={"ID":"3ad876a6-05a6-4977-b658-3752521a5d7c","Type":"ContainerDied","Data":"be6c93a93cff5059b37ab1ce81d39fcfb79b9b85c59595e25b26a6c0b6962ea7"} Jul 24 23:59:03.914327 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.914103 2586 scope.go:117] "RemoveContainer" containerID="2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6" Jul 24 23:59:03.923442 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.923419 2586 scope.go:117] "RemoveContainer" containerID="3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093" Jul 24 23:59:03.931310 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.931293 2586 scope.go:117] "RemoveContainer" containerID="2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6" Jul 24 23:59:03.931536 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:59:03.931520 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6\": container with ID starting with 2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6 not found: ID does not exist" containerID="2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6" Jul 24 23:59:03.931593 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.931543 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6"} err="failed to get container status \"2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6\": rpc error: code = NotFound desc = could not find container \"2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6\": container with ID starting with 2c78d9bc4963cc802bed28820358bb38249341ae68639b6d81ce30b812662bf6 not found: ID does not exist" Jul 24 23:59:03.931593 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.931557 2586 scope.go:117] "RemoveContainer" containerID="3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093" Jul 24 23:59:03.931752 ip-10-0-131-63 kubenswrapper[2586]: E0724 23:59:03.931735 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093\": container with ID starting with 3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093 not found: ID does not exist" containerID="3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093" Jul 24 23:59:03.931791 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.931755 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093"} err="failed to get container status \"3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093\": rpc error: code = NotFound desc = could not find container \"3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093\": container with ID starting with 3dc27a0f095d66f92f0c228e2bcd2635408cab7812d5894735fcba62a89b7093 not found: ID does not exist" Jul 24 23:59:03.936438 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.936414 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl"] Jul 24 23:59:03.940180 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:03.940161 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-9ffa79f3-openshift-default-7c8f7c9fd4-ht8gl"] Jul 24 23:59:04.787040 ip-10-0-131-63 kubenswrapper[2586]: I0724 23:59:04.787010 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" path="/var/lib/kubelet/pods/3ad876a6-05a6-4977-b658-3752521a5d7c/volumes" Jul 25 00:00:00.134708 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.134675 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29748960-5kpcj"] Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135031 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d880556-4bdc-4fb4-86d4-ab0a231ac3d1" containerName="revoke-keys" Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135042 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d880556-4bdc-4fb4-86d4-ab0a231ac3d1" containerName="revoke-keys" Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135063 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135069 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135124 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d880556-4bdc-4fb4-86d4-ab0a231ac3d1" containerName="revoke-keys" Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135133 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" Jul 25 00:00:00.135164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.135142 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" Jul 25 00:00:00.138193 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.138178 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.140591 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.140570 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Jul 25 00:00:00.140728 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.140594 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-w2hvh\"" Jul 25 00:00:00.145624 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.145601 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29748960-5kpcj"] Jul 25 00:00:00.298300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.298263 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqz7\" (UniqueName: \"kubernetes.io/projected/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-kube-api-access-4nqz7\") pod \"image-pruner-29748960-5kpcj\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.298442 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.298402 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-serviceca\") pod \"image-pruner-29748960-5kpcj\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.399634 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.399564 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-serviceca\") pod \"image-pruner-29748960-5kpcj\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.399634 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.399623 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqz7\" (UniqueName: \"kubernetes.io/projected/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-kube-api-access-4nqz7\") pod \"image-pruner-29748960-5kpcj\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.400390 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.400368 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-serviceca\") pod \"image-pruner-29748960-5kpcj\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.406931 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.406898 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqz7\" (UniqueName: \"kubernetes.io/projected/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-kube-api-access-4nqz7\") pod \"image-pruner-29748960-5kpcj\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.464669 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.464641 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:00.596888 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:00.596863 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29748960-5kpcj"] Jul 25 00:00:00.599098 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:00:00.599067 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c6f888_1063_4bb9_9f8a_0bfd0f1223d2.slice/crio-5b243b47bbad7d5498f736db983b53484c00a50189e867c0ff5fb089cf682c92 WatchSource:0}: Error finding container 5b243b47bbad7d5498f736db983b53484c00a50189e867c0ff5fb089cf682c92: Status 404 returned error can't find the container with id 5b243b47bbad7d5498f736db983b53484c00a50189e867c0ff5fb089cf682c92 Jul 25 00:00:01.198624 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:01.198584 2586 generic.go:356] "Generic (PLEG): container finished" podID="13c6f888-1063-4bb9-9f8a-0bfd0f1223d2" containerID="f95452c3ba50828b9d68f02e1f527c3fd293cc65f167545a7a9e30745e7c161f" exitCode=0 Jul 25 00:00:01.199049 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:01.198671 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29748960-5kpcj" event={"ID":"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2","Type":"ContainerDied","Data":"f95452c3ba50828b9d68f02e1f527c3fd293cc65f167545a7a9e30745e7c161f"} Jul 25 00:00:01.199049 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:01.198720 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29748960-5kpcj" event={"ID":"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2","Type":"ContainerStarted","Data":"5b243b47bbad7d5498f736db983b53484c00a50189e867c0ff5fb089cf682c92"} Jul 25 00:00:02.354282 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.354228 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:02.520354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.520272 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-serviceca\") pod \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " Jul 25 00:00:02.520354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.520316 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4nqz7\" (UniqueName: \"kubernetes.io/projected/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-kube-api-access-4nqz7\") pod \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\" (UID: \"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2\") " Jul 25 00:00:02.520720 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.520689 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-serviceca" (OuterVolumeSpecName: "serviceca") pod "13c6f888-1063-4bb9-9f8a-0bfd0f1223d2" (UID: "13c6f888-1063-4bb9-9f8a-0bfd0f1223d2"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:00:02.522357 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.522328 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-kube-api-access-4nqz7" (OuterVolumeSpecName: "kube-api-access-4nqz7") pod "13c6f888-1063-4bb9-9f8a-0bfd0f1223d2" (UID: "13c6f888-1063-4bb9-9f8a-0bfd0f1223d2"). InnerVolumeSpecName "kube-api-access-4nqz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:00:02.621804 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.621771 2586 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-serviceca\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:00:02.621804 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:02.621798 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqz7\" (UniqueName: \"kubernetes.io/projected/13c6f888-1063-4bb9-9f8a-0bfd0f1223d2-kube-api-access-4nqz7\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:00:03.210336 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:03.210305 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29748960-5kpcj" Jul 25 00:00:03.210495 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:03.210298 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29748960-5kpcj" event={"ID":"13c6f888-1063-4bb9-9f8a-0bfd0f1223d2","Type":"ContainerDied","Data":"5b243b47bbad7d5498f736db983b53484c00a50189e867c0ff5fb089cf682c92"} Jul 25 00:00:03.210495 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:03.210461 2586 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b243b47bbad7d5498f736db983b53484c00a50189e867c0ff5fb089cf682c92" Jul 25 00:00:35.349445 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.349361 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5"] Jul 25 00:00:35.349969 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.349952 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" Jul 25 00:00:35.350026 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.349973 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ad876a6-05a6-4977-b658-3752521a5d7c" containerName="istio-proxy" Jul 25 00:00:35.350026 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.349995 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c6f888-1063-4bb9-9f8a-0bfd0f1223d2" containerName="image-pruner" Jul 25 00:00:35.350026 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.350005 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c6f888-1063-4bb9-9f8a-0bfd0f1223d2" containerName="image-pruner" Jul 25 00:00:35.350119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.350092 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c6f888-1063-4bb9-9f8a-0bfd0f1223d2" containerName="image-pruner" Jul 25 00:00:35.353578 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.353549 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.356113 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.356091 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-438308e8-openshift-default-dockercfg-t94lg\"" Jul 25 00:00:35.356260 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.356095 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 25 00:00:35.364181 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.364157 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5"] Jul 25 00:00:35.396892 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.396854 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-token\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397068 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.396895 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfb41c-7924-4b53-a751-e898d9cf0848-istiod-ca-cert\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397068 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.396924 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5w9z\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-kube-api-access-m5w9z\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397068 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.396962 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-certs\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397068 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.397029 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-data\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397222 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.397083 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-socket\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397222 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.397112 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-credential-socket\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397222 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.397186 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-podinfo\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.397361 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.397219 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-envoy\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498070 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498032 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-envoy\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498232 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498099 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-token\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498232 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498122 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfb41c-7924-4b53-a751-e898d9cf0848-istiod-ca-cert\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498232 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498144 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5w9z\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-kube-api-access-m5w9z\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498232 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498179 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-certs\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498520 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498226 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-data\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498520 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498309 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-socket\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498520 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498349 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-credential-socket\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498520 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498398 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-podinfo\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498732 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498707 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-data\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498767 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-certs\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.498968 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498946 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfb41c-7924-4b53-a751-e898d9cf0848-istiod-ca-cert\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.499026 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.498949 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-socket\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.499064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.499028 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-credential-socket\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.500535 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.500510 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-envoy\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.500891 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.500870 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-podinfo\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.505660 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.505635 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-token\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.506227 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.506209 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5w9z\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-kube-api-access-m5w9z\") pod \"e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.666953 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.666915 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:35.809770 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.809746 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5"] Jul 25 00:00:35.812220 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:00:35.812194 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dfb41c_7924_4b53_a751_e898d9cf0848.slice/crio-9efcc4526704b1163d287f4eb386c9c58b3abd2584dfb4e7c693181985615e66 WatchSource:0}: Error finding container 9efcc4526704b1163d287f4eb386c9c58b3abd2584dfb4e7c693181985615e66: Status 404 returned error can't find the container with id 9efcc4526704b1163d287f4eb386c9c58b3abd2584dfb4e7c693181985615e66 Jul 25 00:00:35.814836 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.814806 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:00:35.814905 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.814876 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:00:35.814945 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:35.814904 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:00:36.383192 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:36.383149 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" event={"ID":"f4dfb41c-7924-4b53-a751-e898d9cf0848","Type":"ContainerStarted","Data":"a24853b1e9e6c8d9e416a319996a6836b4db8fbcebcecf200e27c60084942d59"} Jul 25 00:00:36.383687 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:36.383210 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" event={"ID":"f4dfb41c-7924-4b53-a751-e898d9cf0848","Type":"ContainerStarted","Data":"9efcc4526704b1163d287f4eb386c9c58b3abd2584dfb4e7c693181985615e66"} Jul 25 00:00:36.402643 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:36.402594 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podStartSLOduration=1.402579898 podStartE2EDuration="1.402579898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:00:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:00:36.401193422 +0000 UTC m=+1896.094034250" watchObservedRunningTime="2026-07-25 00:00:36.402579898 +0000 UTC m=+1896.095420799" Jul 25 00:00:36.667357 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:36.667274 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:36.668915 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:36.668884 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 25 00:00:36.669043 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:36.668944 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 25 00:00:37.670834 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:37.670795 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:38.392701 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:38.392671 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:38.393654 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:38.393630 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:39.372516 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.372480 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq"] Jul 25 00:00:39.376263 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.376232 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.378626 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.378590 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-438308e8\"" Jul 25 00:00:39.381129 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.381105 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd"] Jul 25 00:00:39.384499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.384480 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.385710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.385687 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq"] Jul 25 00:00:39.386619 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.386599 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-438308e8-dockercfg-vzjpt\"" Jul 25 00:00:39.397501 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.397479 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd"] Jul 25 00:00:39.432704 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.432667 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a607e9c6-c604-4349-9b33-dfd10290b730-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.432874 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.432710 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289affe0-9a08-4a9a-97df-fc436e314984-plugins-config-volume\") pod \"payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.432874 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.432750 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2td96\" (UniqueName: \"kubernetes.io/projected/289affe0-9a08-4a9a-97df-fc436e314984-kube-api-access-2td96\") pod \"payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.433021 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.432897 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpj49\" (UniqueName: \"kubernetes.io/projected/a607e9c6-c604-4349-9b33-dfd10290b730-kube-api-access-lpj49\") pod \"payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.533529 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.533502 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a607e9c6-c604-4349-9b33-dfd10290b730-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.533664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.533535 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289affe0-9a08-4a9a-97df-fc436e314984-plugins-config-volume\") pod \"payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.533664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.533559 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2td96\" (UniqueName: \"kubernetes.io/projected/289affe0-9a08-4a9a-97df-fc436e314984-kube-api-access-2td96\") pod \"payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.533664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.533620 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpj49\" (UniqueName: \"kubernetes.io/projected/a607e9c6-c604-4349-9b33-dfd10290b730-kube-api-access-lpj49\") pod \"payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.534333 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.534305 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a607e9c6-c604-4349-9b33-dfd10290b730-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.534476 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.534337 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289affe0-9a08-4a9a-97df-fc436e314984-plugins-config-volume\") pod \"payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.541299 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.541275 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpj49\" (UniqueName: \"kubernetes.io/projected/a607e9c6-c604-4349-9b33-dfd10290b730-kube-api-access-lpj49\") pod \"payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.541393 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.541312 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2td96\" (UniqueName: \"kubernetes.io/projected/289affe0-9a08-4a9a-97df-fc436e314984-kube-api-access-2td96\") pod \"payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.690131 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.690052 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:39.701275 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.701251 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:39.830815 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:39.830790 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq"] Jul 25 00:00:39.833315 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:00:39.833278 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda607e9c6_c604_4349_9b33_dfd10290b730.slice/crio-5ec8af9f24cca61ab7daad184759af43e4c92c5d622a638ca759e693b8da3511 WatchSource:0}: Error finding container 5ec8af9f24cca61ab7daad184759af43e4c92c5d622a638ca759e693b8da3511: Status 404 returned error can't find the container with id 5ec8af9f24cca61ab7daad184759af43e4c92c5d622a638ca759e693b8da3511 Jul 25 00:00:40.068170 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.068130 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd"] Jul 25 00:00:40.070499 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:00:40.070472 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289affe0_9a08_4a9a_97df_fc436e314984.slice/crio-e4a42ab415cb894fbb0807fd376e89386cdbc8e2be30d94c5365a66bbd622bed WatchSource:0}: Error finding container e4a42ab415cb894fbb0807fd376e89386cdbc8e2be30d94c5365a66bbd622bed: Status 404 returned error can't find the container with id e4a42ab415cb894fbb0807fd376e89386cdbc8e2be30d94c5365a66bbd622bed Jul 25 00:00:40.410320 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.410228 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" event={"ID":"289affe0-9a08-4a9a-97df-fc436e314984","Type":"ContainerStarted","Data":"bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471"} Jul 25 00:00:40.410320 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.410327 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" event={"ID":"289affe0-9a08-4a9a-97df-fc436e314984","Type":"ContainerStarted","Data":"e4a42ab415cb894fbb0807fd376e89386cdbc8e2be30d94c5365a66bbd622bed"} Jul 25 00:00:40.410881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.410357 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:40.411991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.411957 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" event={"ID":"a607e9c6-c604-4349-9b33-dfd10290b730","Type":"ContainerStarted","Data":"a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e"} Jul 25 00:00:40.412109 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.412047 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:00:40.412109 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.412087 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" event={"ID":"a607e9c6-c604-4349-9b33-dfd10290b730","Type":"ContainerStarted","Data":"5ec8af9f24cca61ab7daad184759af43e4c92c5d622a638ca759e693b8da3511"} Jul 25 00:00:40.426367 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.426318 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" podStartSLOduration=1.426303432 podStartE2EDuration="1.426303432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:00:40.42444253 +0000 UTC m=+1900.117283357" watchObservedRunningTime="2026-07-25 00:00:40.426303432 +0000 UTC m=+1900.119144259" Jul 25 00:00:40.438640 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:40.438600 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" podStartSLOduration=1.438585158 podStartE2EDuration="1.438585158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:00:40.437888677 +0000 UTC m=+1900.130729535" watchObservedRunningTime="2026-07-25 00:00:40.438585158 +0000 UTC m=+1900.131425987" Jul 25 00:00:51.419844 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:51.419805 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" podUID="a607e9c6-c604-4349-9b33-dfd10290b730" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.60:9004: connect: connection refused" Jul 25 00:00:51.420636 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:51.420613 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:00:53.394330 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.394292 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 25 00:00:53.394745 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.394349 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 25 00:00:53.491393 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.491364 2586 generic.go:356] "Generic (PLEG): container finished" podID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerID="a24853b1e9e6c8d9e416a319996a6836b4db8fbcebcecf200e27c60084942d59" exitCode=0 Jul 25 00:00:53.491549 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.491437 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" event={"ID":"f4dfb41c-7924-4b53-a751-e898d9cf0848","Type":"ContainerDied","Data":"a24853b1e9e6c8d9e416a319996a6836b4db8fbcebcecf200e27c60084942d59"} Jul 25 00:00:53.491809 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.491795 2586 scope.go:117] "RemoveContainer" containerID="a24853b1e9e6c8d9e416a319996a6836b4db8fbcebcecf200e27c60084942d59" Jul 25 00:00:53.492910 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.492882 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:00:53.493000 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.492966 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:00:53.493039 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:53.493012 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:00:54.498564 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:54.498527 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" event={"ID":"f4dfb41c-7924-4b53-a751-e898d9cf0848","Type":"ContainerStarted","Data":"77cab0515f00da5102f753708ba550066b637a3d414e31e60a2c3df82f55ae13"} Jul 25 00:00:54.667500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:54.667468 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:54.667691 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:54.667669 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 25 00:00:54.667736 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:54.667717 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 25 00:00:55.668065 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:55.668028 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 25 00:00:55.668548 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:55.668089 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 25 00:00:56.667855 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:56.667817 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 25 00:00:56.668041 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:56.667872 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 25 00:00:57.668037 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:57.667999 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 25 00:00:57.668536 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:57.668063 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 25 00:00:58.671974 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:58.671941 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:58.672396 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:58.672302 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:00:58.673011 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:00:58.672990 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:01:00.460279 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.460232 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd"] Jul 25 00:01:00.460687 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.460488 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" podUID="289affe0-9a08-4a9a-97df-fc436e314984" containerName="payload-processing" containerID="cri-o://bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471" gracePeriod=30 Jul 25 00:01:00.467115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.467090 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq"] Jul 25 00:01:00.467349 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.467323 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" podUID="a607e9c6-c604-4349-9b33-dfd10290b730" containerName="payload-pre-processing" containerID="cri-o://a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e" gracePeriod=30 Jul 25 00:01:00.468169 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.468110 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:01:00.718888 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.718827 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:01:00.751312 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.751289 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:01:00.829781 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.829745 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a607e9c6-c604-4349-9b33-dfd10290b730-plugins-config-volume\") pod \"a607e9c6-c604-4349-9b33-dfd10290b730\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " Jul 25 00:01:00.829939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.829786 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2td96\" (UniqueName: \"kubernetes.io/projected/289affe0-9a08-4a9a-97df-fc436e314984-kube-api-access-2td96\") pod \"289affe0-9a08-4a9a-97df-fc436e314984\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " Jul 25 00:01:00.829939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.829869 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289affe0-9a08-4a9a-97df-fc436e314984-plugins-config-volume\") pod \"289affe0-9a08-4a9a-97df-fc436e314984\" (UID: \"289affe0-9a08-4a9a-97df-fc436e314984\") " Jul 25 00:01:00.829939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.829885 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpj49\" (UniqueName: \"kubernetes.io/projected/a607e9c6-c604-4349-9b33-dfd10290b730-kube-api-access-lpj49\") pod \"a607e9c6-c604-4349-9b33-dfd10290b730\" (UID: \"a607e9c6-c604-4349-9b33-dfd10290b730\") " Jul 25 00:01:00.830144 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.830119 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a607e9c6-c604-4349-9b33-dfd10290b730-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a607e9c6-c604-4349-9b33-dfd10290b730" (UID: "a607e9c6-c604-4349-9b33-dfd10290b730"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:01:00.830390 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.830369 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289affe0-9a08-4a9a-97df-fc436e314984-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "289affe0-9a08-4a9a-97df-fc436e314984" (UID: "289affe0-9a08-4a9a-97df-fc436e314984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:01:00.831765 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.831736 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/289affe0-9a08-4a9a-97df-fc436e314984-kube-api-access-2td96" (OuterVolumeSpecName: "kube-api-access-2td96") pod "289affe0-9a08-4a9a-97df-fc436e314984" (UID: "289affe0-9a08-4a9a-97df-fc436e314984"). InnerVolumeSpecName "kube-api-access-2td96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:01:00.831858 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.831837 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a607e9c6-c604-4349-9b33-dfd10290b730-kube-api-access-lpj49" (OuterVolumeSpecName: "kube-api-access-lpj49") pod "a607e9c6-c604-4349-9b33-dfd10290b730" (UID: "a607e9c6-c604-4349-9b33-dfd10290b730"). InnerVolumeSpecName "kube-api-access-lpj49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:01:00.930653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.930630 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a607e9c6-c604-4349-9b33-dfd10290b730-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:00.930653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.930652 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2td96\" (UniqueName: \"kubernetes.io/projected/289affe0-9a08-4a9a-97df-fc436e314984-kube-api-access-2td96\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:00.930788 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.930663 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/289affe0-9a08-4a9a-97df-fc436e314984-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:00.930788 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:00.930673 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpj49\" (UniqueName: \"kubernetes.io/projected/a607e9c6-c604-4349-9b33-dfd10290b730-kube-api-access-lpj49\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:01.534582 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.534546 2586 generic.go:356] "Generic (PLEG): container finished" podID="289affe0-9a08-4a9a-97df-fc436e314984" containerID="bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471" exitCode=0 Jul 25 00:01:01.535087 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.534603 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" Jul 25 00:01:01.535087 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.534634 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" event={"ID":"289affe0-9a08-4a9a-97df-fc436e314984","Type":"ContainerDied","Data":"bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471"} Jul 25 00:01:01.535087 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.534694 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd" event={"ID":"289affe0-9a08-4a9a-97df-fc436e314984","Type":"ContainerDied","Data":"e4a42ab415cb894fbb0807fd376e89386cdbc8e2be30d94c5365a66bbd622bed"} Jul 25 00:01:01.535087 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.534725 2586 scope.go:117] "RemoveContainer" containerID="bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471" Jul 25 00:01:01.536410 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.536383 2586 generic.go:356] "Generic (PLEG): container finished" podID="a607e9c6-c604-4349-9b33-dfd10290b730" containerID="a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e" exitCode=0 Jul 25 00:01:01.536558 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.536435 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" Jul 25 00:01:01.536558 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.536445 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" event={"ID":"a607e9c6-c604-4349-9b33-dfd10290b730","Type":"ContainerDied","Data":"a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e"} Jul 25 00:01:01.536558 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.536493 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq" event={"ID":"a607e9c6-c604-4349-9b33-dfd10290b730","Type":"ContainerDied","Data":"5ec8af9f24cca61ab7daad184759af43e4c92c5d622a638ca759e693b8da3511"} Jul 25 00:01:01.548560 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.548542 2586 scope.go:117] "RemoveContainer" containerID="bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471" Jul 25 00:01:01.548885 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:01:01.548865 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471\": container with ID starting with bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471 not found: ID does not exist" containerID="bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471" Jul 25 00:01:01.548965 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.548894 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471"} err="failed to get container status \"bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471\": rpc error: code = NotFound desc = could not find container \"bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471\": container with ID starting with bd679fc99916b74fe48074fe3c97cbd1649a03938ecad8bb70ddf3c29176c471 not found: ID does not exist" Jul 25 00:01:01.548965 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.548915 2586 scope.go:117] "RemoveContainer" containerID="a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e" Jul 25 00:01:01.557140 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.557122 2586 scope.go:117] "RemoveContainer" containerID="a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e" Jul 25 00:01:01.557418 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:01:01.557395 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e\": container with ID starting with a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e not found: ID does not exist" containerID="a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e" Jul 25 00:01:01.557480 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.557428 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e"} err="failed to get container status \"a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e\": rpc error: code = NotFound desc = could not find container \"a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e\": container with ID starting with a03fa04f8538caca254253d6500fd5ab5fac32e09850e7fcbd4364f07fd3d32e not found: ID does not exist" Jul 25 00:01:01.559786 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.559765 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd"] Jul 25 00:01:01.563468 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.563449 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-438308e8-6d95f6d8c4-c8cnd"] Jul 25 00:01:01.572144 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.572124 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq"] Jul 25 00:01:01.575756 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:01.575737 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-438308e8-59ff75bb8d-bnwnq"] Jul 25 00:01:02.787794 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:02.787761 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289affe0-9a08-4a9a-97df-fc436e314984" path="/var/lib/kubelet/pods/289affe0-9a08-4a9a-97df-fc436e314984/volumes" Jul 25 00:01:02.788173 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:02.788121 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a607e9c6-c604-4349-9b33-dfd10290b730" path="/var/lib/kubelet/pods/a607e9c6-c604-4349-9b33-dfd10290b730/volumes" Jul 25 00:01:11.416054 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:11.416016 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5"] Jul 25 00:01:11.416610 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:11.416277 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" containerID="cri-o://77cab0515f00da5102f753708ba550066b637a3d414e31e60a2c3df82f55ae13" gracePeriod=30 Jul 25 00:01:13.674560 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:13.674523 2586 patch_prober.go:28] interesting pod/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 25 00:01:13.674935 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:13.674593 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 25 00:01:16.620432 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.620399 2586 generic.go:356] "Generic (PLEG): container finished" podID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerID="77cab0515f00da5102f753708ba550066b637a3d414e31e60a2c3df82f55ae13" exitCode=0 Jul 25 00:01:16.620784 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.620461 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" event={"ID":"f4dfb41c-7924-4b53-a751-e898d9cf0848","Type":"ContainerDied","Data":"77cab0515f00da5102f753708ba550066b637a3d414e31e60a2c3df82f55ae13"} Jul 25 00:01:16.620784 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.620512 2586 scope.go:117] "RemoveContainer" containerID="a24853b1e9e6c8d9e416a319996a6836b4db8fbcebcecf200e27c60084942d59" Jul 25 00:01:16.664818 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.664796 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:01:16.862303 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862281 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-certs\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862470 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862316 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-credential-socket\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862470 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862345 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-token\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862470 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862363 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-socket\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862470 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862390 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-podinfo\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862685 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862555 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfb41c-7924-4b53-a751-e898d9cf0848-istiod-ca-cert\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862685 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862558 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:01:16.862685 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862605 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-data\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862685 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862603 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:01:16.862882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862687 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5w9z\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-kube-api-access-m5w9z\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.862882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862709 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:01:16.862882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862720 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-envoy\") pod \"f4dfb41c-7924-4b53-a751-e898d9cf0848\" (UID: \"f4dfb41c-7924-4b53-a751-e898d9cf0848\") " Jul 25 00:01:16.863030 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.862922 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dfb41c-7924-4b53-a751-e898d9cf0848-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:01:16.863091 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.863057 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.863091 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.863075 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.863199 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.863090 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.863199 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.863103 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4dfb41c-7924-4b53-a751-e898d9cf0848-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.863464 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.863446 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:01:16.865164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.865139 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-kube-api-access-m5w9z" (OuterVolumeSpecName: "kube-api-access-m5w9z") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "kube-api-access-m5w9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:01:16.865283 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.865174 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:01:16.865283 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.865190 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:01:16.865283 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.865193 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4dfb41c-7924-4b53-a751-e898d9cf0848" (UID: "f4dfb41c-7924-4b53-a751-e898d9cf0848"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:01:16.963496 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.963473 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.963496 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.963495 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.963632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.963504 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.963632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.963514 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5w9z\" (UniqueName: \"kubernetes.io/projected/f4dfb41c-7924-4b53-a751-e898d9cf0848-kube-api-access-m5w9z\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:16.963632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:16.963522 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4dfb41c-7924-4b53-a751-e898d9cf0848-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:01:17.627902 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:17.627867 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" Jul 25 00:01:17.628436 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:17.627870 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5" event={"ID":"f4dfb41c-7924-4b53-a751-e898d9cf0848","Type":"ContainerDied","Data":"9efcc4526704b1163d287f4eb386c9c58b3abd2584dfb4e7c693181985615e66"} Jul 25 00:01:17.628436 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:17.628035 2586 scope.go:117] "RemoveContainer" containerID="77cab0515f00da5102f753708ba550066b637a3d414e31e60a2c3df82f55ae13" Jul 25 00:01:17.649693 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:17.649658 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5"] Jul 25 00:01:17.654979 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:17.654948 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-438308e8-openshift-default-7c67f96dc6-g4jw5"] Jul 25 00:01:18.786524 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:18.786492 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" path="/var/lib/kubelet/pods/f4dfb41c-7924-4b53-a751-e898d9cf0848/volumes" Jul 25 00:01:30.926317 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926271 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885"] Jul 25 00:01:30.926849 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926833 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" Jul 25 00:01:30.926917 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926854 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" Jul 25 00:01:30.926917 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926865 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" Jul 25 00:01:30.926917 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926874 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" Jul 25 00:01:30.926917 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926906 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a607e9c6-c604-4349-9b33-dfd10290b730" containerName="payload-pre-processing" Jul 25 00:01:30.926917 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926915 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a607e9c6-c604-4349-9b33-dfd10290b730" containerName="payload-pre-processing" Jul 25 00:01:30.927089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926930 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289affe0-9a08-4a9a-97df-fc436e314984" containerName="payload-processing" Jul 25 00:01:30.927089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.926938 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="289affe0-9a08-4a9a-97df-fc436e314984" containerName="payload-processing" Jul 25 00:01:30.927089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.927022 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a607e9c6-c604-4349-9b33-dfd10290b730" containerName="payload-pre-processing" Jul 25 00:01:30.927089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.927033 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="289affe0-9a08-4a9a-97df-fc436e314984" containerName="payload-processing" Jul 25 00:01:30.927089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.927046 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" Jul 25 00:01:30.927089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.927055 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4dfb41c-7924-4b53-a751-e898d9cf0848" containerName="istio-proxy" Jul 25 00:01:30.930550 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.930527 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.933380 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.933356 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-292ac4d2-openshift-default-dockercfg-2gbqg\"" Jul 25 00:01:30.933509 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.933361 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 25 00:01:30.941165 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.941132 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885"] Jul 25 00:01:30.981960 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.981922 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m65t\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-kube-api-access-6m65t\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982163 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.981968 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42991137-a078-47da-88e4-d672eff6f493-istio-podinfo\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982163 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982068 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-istio-token\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982163 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982110 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42991137-a078-47da-88e4-d672eff6f493-istiod-ca-cert\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982178 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-credential-socket\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982199 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-certs\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982220 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-envoy\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982261 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-data\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:30.982402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:30.982316 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-socket\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083348 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083314 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42991137-a078-47da-88e4-d672eff6f493-istiod-ca-cert\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083367 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-credential-socket\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083386 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-certs\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083408 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-envoy\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083432 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-data\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083472 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-socket\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083797 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083529 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m65t\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-kube-api-access-6m65t\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083797 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083556 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42991137-a078-47da-88e4-d672eff6f493-istio-podinfo\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083797 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083601 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-istio-token\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083962 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083824 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-certs\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.083962 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.083884 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-credential-socket\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.084170 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.084137 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-socket\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.084307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.084281 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-data\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.084420 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.084287 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42991137-a078-47da-88e4-d672eff6f493-istiod-ca-cert\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.086392 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.086361 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42991137-a078-47da-88e4-d672eff6f493-istio-podinfo\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.086482 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.086396 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-envoy\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.091569 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.091539 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m65t\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-kube-api-access-6m65t\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.091569 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.091559 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-istio-token\") pod \"e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.245104 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.245004 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:31.398276 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.398246 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885"] Jul 25 00:01:31.401194 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:01:31.401165 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42991137_a078_47da_88e4_d672eff6f493.slice/crio-487060e7cdef15899c1fbf02e5215cfe6b617456d785b7b5057660340dd3d565 WatchSource:0}: Error finding container 487060e7cdef15899c1fbf02e5215cfe6b617456d785b7b5057660340dd3d565: Status 404 returned error can't find the container with id 487060e7cdef15899c1fbf02e5215cfe6b617456d785b7b5057660340dd3d565 Jul 25 00:01:31.403402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.403368 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:01:31.403497 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.403441 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:01:31.403497 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.403471 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:01:31.701495 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.701460 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" event={"ID":"42991137-a078-47da-88e4-d672eff6f493","Type":"ContainerStarted","Data":"662f39600c26b730f33233e8eddf5da19d4d095f32c93fed04941e7b071a8e59"} Jul 25 00:01:31.701718 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.701510 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" event={"ID":"42991137-a078-47da-88e4-d672eff6f493","Type":"ContainerStarted","Data":"487060e7cdef15899c1fbf02e5215cfe6b617456d785b7b5057660340dd3d565"} Jul 25 00:01:31.722090 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:31.722033 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podStartSLOduration=1.722017371 podStartE2EDuration="1.722017371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:01:31.719131685 +0000 UTC m=+1951.411972516" watchObservedRunningTime="2026-07-25 00:01:31.722017371 +0000 UTC m=+1951.414858271" Jul 25 00:01:32.245403 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:32.245358 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:32.250668 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:32.250635 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:32.707335 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:32.707299 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:32.708626 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:32.708604 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:34.475913 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.475878 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5"] Jul 25 00:01:34.478647 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.478619 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.480994 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.480971 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-292ac4d2\"" Jul 25 00:01:34.481159 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.481127 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-292ac4d2-dockercfg-5qg5b\"" Jul 25 00:01:34.487970 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.487945 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5"] Jul 25 00:01:34.514061 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.514033 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-plugins-config-volume\") pod \"payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.514170 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.514080 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg6gj\" (UniqueName: \"kubernetes.io/projected/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-kube-api-access-hg6gj\") pod \"payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.614573 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.614539 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg6gj\" (UniqueName: \"kubernetes.io/projected/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-kube-api-access-hg6gj\") pod \"payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.614734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.614658 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-plugins-config-volume\") pod \"payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.615343 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.615326 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-plugins-config-volume\") pod \"payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.628423 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.628384 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg6gj\" (UniqueName: \"kubernetes.io/projected/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-kube-api-access-hg6gj\") pod \"payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.790775 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.790700 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:34.921866 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:34.921842 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5"] Jul 25 00:01:34.924343 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:01:34.924315 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77237bbb_3d6d_4fd4_b736_2bff0df3aae3.slice/crio-75452b5e2480d61f22b5a72061cba14b6c07868183206ba32d31278e22f62f8e WatchSource:0}: Error finding container 75452b5e2480d61f22b5a72061cba14b6c07868183206ba32d31278e22f62f8e: Status 404 returned error can't find the container with id 75452b5e2480d61f22b5a72061cba14b6c07868183206ba32d31278e22f62f8e Jul 25 00:01:35.724008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:35.723965 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" event={"ID":"77237bbb-3d6d-4fd4-b736-2bff0df3aae3","Type":"ContainerStarted","Data":"ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62"} Jul 25 00:01:35.724431 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:35.724037 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" event={"ID":"77237bbb-3d6d-4fd4-b736-2bff0df3aae3","Type":"ContainerStarted","Data":"75452b5e2480d61f22b5a72061cba14b6c07868183206ba32d31278e22f62f8e"} Jul 25 00:01:35.724431 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:35.724141 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:35.739729 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:35.739684 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" podStartSLOduration=1.7396709320000001 podStartE2EDuration="1.739670932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:01:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:01:35.73814664 +0000 UTC m=+1955.430987470" watchObservedRunningTime="2026-07-25 00:01:35.739670932 +0000 UTC m=+1955.432511760" Jul 25 00:01:46.731047 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.731011 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:01:46.791515 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.791477 2586 generic.go:356] "Generic (PLEG): container finished" podID="42991137-a078-47da-88e4-d672eff6f493" containerID="662f39600c26b730f33233e8eddf5da19d4d095f32c93fed04941e7b071a8e59" exitCode=0 Jul 25 00:01:46.791515 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.791517 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" event={"ID":"42991137-a078-47da-88e4-d672eff6f493","Type":"ContainerDied","Data":"662f39600c26b730f33233e8eddf5da19d4d095f32c93fed04941e7b071a8e59"} Jul 25 00:01:46.791960 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.791940 2586 scope.go:117] "RemoveContainer" containerID="662f39600c26b730f33233e8eddf5da19d4d095f32c93fed04941e7b071a8e59" Jul 25 00:01:46.793095 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.793066 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:01:46.793161 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.793138 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:01:46.793209 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:46.793198 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:01:47.707744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:47.707692 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:47.801116 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:47.801085 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" event={"ID":"42991137-a078-47da-88e4-d672eff6f493","Type":"ContainerStarted","Data":"ef5dceccefa1bee953aa92d3d6579a673cc19cf995a32ed130684e6f0433e5ef"} Jul 25 00:01:48.245833 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:48.245746 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:48.247442 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:48.247083 2586 patch_prober.go:28] interesting pod/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 25 00:01:48.247442 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:48.247152 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 25 00:01:49.245894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:49.245463 2586 patch_prober.go:28] interesting pod/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 25 00:01:49.245894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:49.245523 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 25 00:01:50.246522 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:50.246480 2586 patch_prober.go:28] interesting pod/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 25 00:01:50.247016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:50.246551 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 25 00:01:51.245732 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:51.245695 2586 patch_prober.go:28] interesting pod/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 25 00:01:51.246044 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:51.246012 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 25 00:01:52.249604 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:52.249568 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:52.250059 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:52.249972 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:52.250572 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:52.250552 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:01:56.693409 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.693320 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5"] Jul 25 00:01:56.696334 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.696312 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.699548 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.699519 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-292ac4d2\"/\"e2e-lifecycle-model-292ac4d2-kserve-self-signed-certs\"" Jul 25 00:01:56.699548 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.699542 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-292ac4d2\"/\"default-dockercfg-9v82r\"" Jul 25 00:01:56.699739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.699556 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-292ac4d2\"/\"kube-root-ca.crt\"" Jul 25 00:01:56.699739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.699542 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-292ac4d2\"/\"openshift-service-ca.crt\"" Jul 25 00:01:56.704995 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.704976 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5"] Jul 25 00:01:56.802082 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.802053 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tmp-dir\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.802220 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.802085 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tls-certs\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.802220 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.802177 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-dshm\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.802220 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.802206 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-model-cache\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.802364 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.802232 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-946dj\" (UniqueName: \"kubernetes.io/projected/d5473b4f-6c55-42c5-9594-b42287ec8fe6-kube-api-access-946dj\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.802364 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.802294 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-home\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903303 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903274 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-dshm\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903303 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903305 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-model-cache\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903539 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903330 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-946dj\" (UniqueName: \"kubernetes.io/projected/d5473b4f-6c55-42c5-9594-b42287ec8fe6-kube-api-access-946dj\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903539 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903457 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-home\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903539 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903515 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tmp-dir\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903703 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903542 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tls-certs\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903762 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903741 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-model-cache\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903843 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903817 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-home\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.903900 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.903845 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tmp-dir\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.905561 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.905535 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-dshm\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.905950 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.905930 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tls-certs\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:56.910659 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:56.910632 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-946dj\" (UniqueName: \"kubernetes.io/projected/d5473b4f-6c55-42c5-9594-b42287ec8fe6-kube-api-access-946dj\") pod \"e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:57.007732 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:57.007671 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:01:57.143859 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:57.143678 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5"] Jul 25 00:01:57.146343 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:01:57.146312 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5473b4f_6c55_42c5_9594_b42287ec8fe6.slice/crio-57cc75b87fd0eccfef7052045bcdf974a3fba472fc69a38252be561c0ea5d56b WatchSource:0}: Error finding container 57cc75b87fd0eccfef7052045bcdf974a3fba472fc69a38252be561c0ea5d56b: Status 404 returned error can't find the container with id 57cc75b87fd0eccfef7052045bcdf974a3fba472fc69a38252be561c0ea5d56b Jul 25 00:01:57.852319 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:57.852280 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" event={"ID":"d5473b4f-6c55-42c5-9594-b42287ec8fe6","Type":"ContainerStarted","Data":"2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1"} Jul 25 00:01:57.852691 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:57.852325 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" event={"ID":"d5473b4f-6c55-42c5-9594-b42287ec8fe6","Type":"ContainerStarted","Data":"57cc75b87fd0eccfef7052045bcdf974a3fba472fc69a38252be561c0ea5d56b"} Jul 25 00:01:57.868921 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:01:57.868871 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" podStartSLOduration=1.86885721 podStartE2EDuration="1.86885721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:01:57.866958987 +0000 UTC m=+1977.559799815" watchObservedRunningTime="2026-07-25 00:01:57.86885721 +0000 UTC m=+1977.561698037" Jul 25 00:02:07.008470 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:07.008432 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:02:07.008905 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:07.008481 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:02:07.021067 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:07.021044 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:02:07.920169 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:07.920140 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:02:38.557935 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.557899 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5"] Jul 25 00:02:38.558496 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.558139 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" podUID="77237bbb-3d6d-4fd4-b736-2bff0df3aae3" containerName="payload-processing" containerID="cri-o://ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62" gracePeriod=30 Jul 25 00:02:38.816768 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.816709 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:02:38.945835 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.945796 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-plugins-config-volume\") pod \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " Jul 25 00:02:38.945835 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.945842 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hg6gj\" (UniqueName: \"kubernetes.io/projected/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-kube-api-access-hg6gj\") pod \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\" (UID: \"77237bbb-3d6d-4fd4-b736-2bff0df3aae3\") " Jul 25 00:02:38.946187 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.946164 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "77237bbb-3d6d-4fd4-b736-2bff0df3aae3" (UID: "77237bbb-3d6d-4fd4-b736-2bff0df3aae3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:02:38.948057 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:38.948016 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-kube-api-access-hg6gj" (OuterVolumeSpecName: "kube-api-access-hg6gj") pod "77237bbb-3d6d-4fd4-b736-2bff0df3aae3" (UID: "77237bbb-3d6d-4fd4-b736-2bff0df3aae3"). InnerVolumeSpecName "kube-api-access-hg6gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:02:39.046696 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.046656 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:02:39.046696 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.046696 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg6gj\" (UniqueName: \"kubernetes.io/projected/77237bbb-3d6d-4fd4-b736-2bff0df3aae3-kube-api-access-hg6gj\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:02:39.081911 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.081834 2586 generic.go:356] "Generic (PLEG): container finished" podID="77237bbb-3d6d-4fd4-b736-2bff0df3aae3" containerID="ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62" exitCode=0 Jul 25 00:02:39.081911 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.081872 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" Jul 25 00:02:39.082096 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.081906 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" event={"ID":"77237bbb-3d6d-4fd4-b736-2bff0df3aae3","Type":"ContainerDied","Data":"ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62"} Jul 25 00:02:39.082096 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.081958 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5" event={"ID":"77237bbb-3d6d-4fd4-b736-2bff0df3aae3","Type":"ContainerDied","Data":"75452b5e2480d61f22b5a72061cba14b6c07868183206ba32d31278e22f62f8e"} Jul 25 00:02:39.082096 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.081981 2586 scope.go:117] "RemoveContainer" containerID="ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62" Jul 25 00:02:39.091012 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.090992 2586 scope.go:117] "RemoveContainer" containerID="ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62" Jul 25 00:02:39.091281 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:02:39.091261 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62\": container with ID starting with ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62 not found: ID does not exist" containerID="ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62" Jul 25 00:02:39.091361 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.091288 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62"} err="failed to get container status \"ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62\": rpc error: code = NotFound desc = could not find container \"ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62\": container with ID starting with ad986a9478e732a28bae7c9d60e182a178bbf84ff207ff53d4a0898d828e2c62 not found: ID does not exist" Jul 25 00:02:39.102226 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.102195 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5"] Jul 25 00:02:39.104156 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:39.104132 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-292ac4d2-7d6d9944bb-zt7l5"] Jul 25 00:02:40.787190 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:40.787152 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77237bbb-3d6d-4fd4-b736-2bff0df3aae3" path="/var/lib/kubelet/pods/77237bbb-3d6d-4fd4-b736-2bff0df3aae3/volumes" Jul 25 00:02:51.562596 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:02:51.562557 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5"] Jul 25 00:03:01.017575 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:01.017544 2586 scope.go:117] "RemoveContainer" containerID="32555cfc3a9f71552f0c2a11f8f0b9a35bde85ffa273d39e227f8481c77b92f0" Jul 25 00:03:06.582521 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.582484 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" podUID="d5473b4f-6c55-42c5-9594-b42287ec8fe6" containerName="main" containerID="cri-o://2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1" gracePeriod=45 Jul 25 00:03:06.836633 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.836575 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:03:06.991335 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991305 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-946dj\" (UniqueName: \"kubernetes.io/projected/d5473b4f-6c55-42c5-9594-b42287ec8fe6-kube-api-access-946dj\") pod \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " Jul 25 00:03:06.991504 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991357 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tmp-dir\") pod \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " Jul 25 00:03:06.991504 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991391 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-model-cache\") pod \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " Jul 25 00:03:06.991585 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991504 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-dshm\") pod \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " Jul 25 00:03:06.991585 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991554 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tls-certs\") pod \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " Jul 25 00:03:06.991666 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991607 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5473b4f-6c55-42c5-9594-b42287ec8fe6" (UID: "d5473b4f-6c55-42c5-9594-b42287ec8fe6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:06.991666 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991619 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-home\") pod \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\" (UID: \"d5473b4f-6c55-42c5-9594-b42287ec8fe6\") " Jul 25 00:03:06.991666 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991637 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5473b4f-6c55-42c5-9594-b42287ec8fe6" (UID: "d5473b4f-6c55-42c5-9594-b42287ec8fe6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:06.991910 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991888 2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tmp-dir\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:06.992037 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991923 2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-model-cache\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:06.992037 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.991932 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-home" (OuterVolumeSpecName: "home") pod "d5473b4f-6c55-42c5-9594-b42287ec8fe6" (UID: "d5473b4f-6c55-42c5-9594-b42287ec8fe6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:06.993742 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.993719 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5473b4f-6c55-42c5-9594-b42287ec8fe6" (UID: "d5473b4f-6c55-42c5-9594-b42287ec8fe6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 00:03:06.993980 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.993954 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-dshm" (OuterVolumeSpecName: "dshm") pod "d5473b4f-6c55-42c5-9594-b42287ec8fe6" (UID: "d5473b4f-6c55-42c5-9594-b42287ec8fe6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:06.993980 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:06.993971 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5473b4f-6c55-42c5-9594-b42287ec8fe6-kube-api-access-946dj" (OuterVolumeSpecName: "kube-api-access-946dj") pod "d5473b4f-6c55-42c5-9594-b42287ec8fe6" (UID: "d5473b4f-6c55-42c5-9594-b42287ec8fe6"). InnerVolumeSpecName "kube-api-access-946dj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:03:07.092667 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.092611 2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-dshm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:07.092667 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.092632 2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5473b4f-6c55-42c5-9594-b42287ec8fe6-tls-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:07.092667 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.092643 2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5473b4f-6c55-42c5-9594-b42287ec8fe6-home\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:07.092667 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.092651 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-946dj\" (UniqueName: \"kubernetes.io/projected/d5473b4f-6c55-42c5-9594-b42287ec8fe6-kube-api-access-946dj\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:07.230093 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.230066 2586 generic.go:356] "Generic (PLEG): container finished" podID="d5473b4f-6c55-42c5-9594-b42287ec8fe6" containerID="2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1" exitCode=0 Jul 25 00:03:07.230216 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.230125 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" Jul 25 00:03:07.230216 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.230157 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" event={"ID":"d5473b4f-6c55-42c5-9594-b42287ec8fe6","Type":"ContainerDied","Data":"2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1"} Jul 25 00:03:07.230216 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.230204 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5" event={"ID":"d5473b4f-6c55-42c5-9594-b42287ec8fe6","Type":"ContainerDied","Data":"57cc75b87fd0eccfef7052045bcdf974a3fba472fc69a38252be561c0ea5d56b"} Jul 25 00:03:07.230368 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.230227 2586 scope.go:117] "RemoveContainer" containerID="2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1" Jul 25 00:03:07.240018 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.240001 2586 scope.go:117] "RemoveContainer" containerID="2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1" Jul 25 00:03:07.240326 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:03:07.240307 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1\": container with ID starting with 2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1 not found: ID does not exist" containerID="2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1" Jul 25 00:03:07.240398 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.240333 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1"} err="failed to get container status \"2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1\": rpc error: code = NotFound desc = could not find container \"2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1\": container with ID starting with 2b3f246eb8d9b27caa0afdb3f43efbb0c7ac674aa270be3b365fedbc6f1df1a1 not found: ID does not exist" Jul 25 00:03:07.251015 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.250986 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5"] Jul 25 00:03:07.252830 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:07.252806 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-292ac4d2/e2e-lifecycle-model-292ac4d2-kserve-5477f544dc-pl2q5"] Jul 25 00:03:08.786100 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:08.786065 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5473b4f-6c55-42c5-9594-b42287ec8fe6" path="/var/lib/kubelet/pods/d5473b4f-6c55-42c5-9594-b42287ec8fe6/volumes" Jul 25 00:03:34.214628 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:34.214522 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885"] Jul 25 00:03:34.215174 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:34.214805 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" containerID="cri-o://ef5dceccefa1bee953aa92d3d6579a673cc19cf995a32ed130684e6f0433e5ef" gracePeriod=30 Jul 25 00:03:37.252124 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:37.252075 2586 patch_prober.go:28] interesting pod/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 25 00:03:37.252523 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:37.252139 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 25 00:03:39.401228 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.401196 2586 generic.go:356] "Generic (PLEG): container finished" podID="42991137-a078-47da-88e4-d672eff6f493" containerID="ef5dceccefa1bee953aa92d3d6579a673cc19cf995a32ed130684e6f0433e5ef" exitCode=0 Jul 25 00:03:39.401666 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.401274 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" event={"ID":"42991137-a078-47da-88e4-d672eff6f493","Type":"ContainerDied","Data":"ef5dceccefa1bee953aa92d3d6579a673cc19cf995a32ed130684e6f0433e5ef"} Jul 25 00:03:39.401666 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.401341 2586 scope.go:117] "RemoveContainer" containerID="662f39600c26b730f33233e8eddf5da19d4d095f32c93fed04941e7b071a8e59" Jul 25 00:03:39.483230 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.483145 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:03:39.570485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570450 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42991137-a078-47da-88e4-d672eff6f493-istiod-ca-cert\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.570485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570485 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-data\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.570744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570525 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m65t\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-kube-api-access-6m65t\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.570744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570550 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-istio-token\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.570744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570575 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-certs\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.570744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570614 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-socket\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.570744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570649 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-credential-socket\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.571010 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570801 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42991137-a078-47da-88e4-d672eff6f493-istio-podinfo\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.571010 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570831 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-envoy\") pod \"42991137-a078-47da-88e4-d672eff6f493\" (UID: \"42991137-a078-47da-88e4-d672eff6f493\") " Jul 25 00:03:39.571010 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.570993 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:39.571170 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571039 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42991137-a078-47da-88e4-d672eff6f493-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:03:39.571170 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571111 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:39.571323 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571209 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.571323 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571306 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:39.571445 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571230 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.571445 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571375 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42991137-a078-47da-88e4-d672eff6f493-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.571445 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.571414 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-data" (OuterVolumeSpecName: "istio-data") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:39.572889 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.572856 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-kube-api-access-6m65t" (OuterVolumeSpecName: "kube-api-access-6m65t") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "kube-api-access-6m65t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:03:39.573020 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.572963 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42991137-a078-47da-88e4-d672eff6f493-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:03:39.573156 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.573137 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-istio-token" (OuterVolumeSpecName: "istio-token") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:03:39.573272 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.573248 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "42991137-a078-47da-88e4-d672eff6f493" (UID: "42991137-a078-47da-88e4-d672eff6f493"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:03:39.672016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.671980 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m65t\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-kube-api-access-6m65t\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.672016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.672012 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42991137-a078-47da-88e4-d672eff6f493-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.672016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.672021 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.672291 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.672037 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42991137-a078-47da-88e4-d672eff6f493-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.672291 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.672047 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:39.672291 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:39.672054 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42991137-a078-47da-88e4-d672eff6f493-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:03:40.408158 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:40.408125 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" Jul 25 00:03:40.408649 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:40.408176 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885" event={"ID":"42991137-a078-47da-88e4-d672eff6f493","Type":"ContainerDied","Data":"487060e7cdef15899c1fbf02e5215cfe6b617456d785b7b5057660340dd3d565"} Jul 25 00:03:40.408649 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:40.408276 2586 scope.go:117] "RemoveContainer" containerID="ef5dceccefa1bee953aa92d3d6579a673cc19cf995a32ed130684e6f0433e5ef" Jul 25 00:03:40.429172 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:40.429144 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885"] Jul 25 00:03:40.432880 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:40.432857 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-292ac4d2-openshift-default-56bc7d74b8-5j885"] Jul 25 00:03:40.786896 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:03:40.786815 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42991137-a078-47da-88e4-d672eff6f493" path="/var/lib/kubelet/pods/42991137-a078-47da-88e4-d672eff6f493/volumes" Jul 25 00:04:00.926503 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:00.926467 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:04:00.929097 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:00.929070 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:04:29.413053 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413015 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq"] Jul 25 00:04:29.413589 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413574 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" Jul 25 00:04:29.413638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413592 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" Jul 25 00:04:29.413638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413612 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77237bbb-3d6d-4fd4-b736-2bff0df3aae3" containerName="payload-processing" Jul 25 00:04:29.413638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413621 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="77237bbb-3d6d-4fd4-b736-2bff0df3aae3" containerName="payload-processing" Jul 25 00:04:29.413638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413632 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5473b4f-6c55-42c5-9594-b42287ec8fe6" containerName="main" Jul 25 00:04:29.413760 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413641 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5473b4f-6c55-42c5-9594-b42287ec8fe6" containerName="main" Jul 25 00:04:29.413792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413766 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" Jul 25 00:04:29.413792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413779 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="77237bbb-3d6d-4fd4-b736-2bff0df3aae3" containerName="payload-processing" Jul 25 00:04:29.413792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413790 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" Jul 25 00:04:29.413883 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.413799 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5473b4f-6c55-42c5-9594-b42287ec8fe6" containerName="main" Jul 25 00:04:29.417104 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.417087 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.420057 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.420036 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 25 00:04:29.420057 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.420043 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-820f12-openshift-default-dockercfg-xmvrq\"" Jul 25 00:04:29.426859 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.426835 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq"] Jul 25 00:04:29.604154 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604119 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-token\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604386 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604162 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-credential-socket\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604386 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604214 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-podinfo\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604386 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604233 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istiod-ca-cert\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604386 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604274 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-socket\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604386 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604293 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-envoy\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604389 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7plf\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-kube-api-access-d7plf\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604417 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-certs\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.604608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.604434 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-data\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705341 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705211 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-podinfo\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705341 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705291 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istiod-ca-cert\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705341 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705317 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-socket\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705631 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705350 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-envoy\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705631 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705397 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7plf\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-kube-api-access-d7plf\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705631 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705421 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-certs\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705631 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705445 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-data\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705631 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705507 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-token\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705631 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705539 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-credential-socket\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705940 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705742 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-socket\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.705940 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705788 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-certs\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.706047 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.705993 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-credential-socket\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.706047 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.706025 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-data\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.706207 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.706190 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istiod-ca-cert\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.708193 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.708173 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-envoy\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.708559 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.708526 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-podinfo\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.713128 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.713101 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-token\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.713231 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.713179 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7plf\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-kube-api-access-d7plf\") pod \"e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.728048 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.728009 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:29.895153 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.895111 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:29.895321 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.895197 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:29.895321 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.895263 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:29.895759 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:29.895728 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq"] Jul 25 00:04:30.659609 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:30.659571 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" event={"ID":"ee035886-6ae0-46b9-a5c2-5d3fbcad264a","Type":"ContainerStarted","Data":"215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff"} Jul 25 00:04:30.660036 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:30.659629 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" event={"ID":"ee035886-6ae0-46b9-a5c2-5d3fbcad264a","Type":"ContainerStarted","Data":"cccd07aa4f1968bdac7d17db0a93dacd58430209cd80cded5837f34ec32ee177"} Jul 25 00:04:30.679842 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:30.679778 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podStartSLOduration=1.6797565269999999 podStartE2EDuration="1.679756527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:04:30.67778362 +0000 UTC m=+2130.370624449" watchObservedRunningTime="2026-07-25 00:04:30.679756527 +0000 UTC m=+2130.372597355" Jul 25 00:04:30.728632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:30.728590 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:30.730294 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:30.730257 2586 patch_prober.go:28] interesting pod/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 25 00:04:30.730435 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:30.730308 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 25 00:04:31.732127 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:31.732082 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:32.669740 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:32.669709 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:32.670769 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:32.670747 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:33.414309 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.414275 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92"] Jul 25 00:04:33.414886 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.414867 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" Jul 25 00:04:33.414938 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.414891 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="42991137-a078-47da-88e4-d672eff6f493" containerName="istio-proxy" Jul 25 00:04:33.418673 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.418654 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.421070 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.421048 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-820f12\"" Jul 25 00:04:33.421202 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.421067 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-820f12-dockercfg-wp5l9\"" Jul 25 00:04:33.426058 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.426032 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92"] Jul 25 00:04:33.440692 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.440658 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54aef27f-be55-456d-a798-62a090ada7ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-820f12-cf68794d8-62r92\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.440849 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.440784 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w6vg\" (UniqueName: \"kubernetes.io/projected/54aef27f-be55-456d-a798-62a090ada7ca-kube-api-access-5w6vg\") pod \"payload-processing-e2e-inf-a-820f12-cf68794d8-62r92\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.541576 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.541527 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w6vg\" (UniqueName: \"kubernetes.io/projected/54aef27f-be55-456d-a798-62a090ada7ca-kube-api-access-5w6vg\") pod \"payload-processing-e2e-inf-a-820f12-cf68794d8-62r92\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.541763 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.541620 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54aef27f-be55-456d-a798-62a090ada7ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-820f12-cf68794d8-62r92\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.542368 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.542344 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54aef27f-be55-456d-a798-62a090ada7ca-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-820f12-cf68794d8-62r92\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.549519 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.549491 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w6vg\" (UniqueName: \"kubernetes.io/projected/54aef27f-be55-456d-a798-62a090ada7ca-kube-api-access-5w6vg\") pod \"payload-processing-e2e-inf-a-820f12-cf68794d8-62r92\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.731804 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.731711 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:33.882293 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:33.882265 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92"] Jul 25 00:04:33.885046 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:04:33.885016 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54aef27f_be55_456d_a798_62a090ada7ca.slice/crio-d2ddf67a316c000bc7f589bc30f1979e36971defd9e10d88a56b50a3dd8b26c5 WatchSource:0}: Error finding container d2ddf67a316c000bc7f589bc30f1979e36971defd9e10d88a56b50a3dd8b26c5: Status 404 returned error can't find the container with id d2ddf67a316c000bc7f589bc30f1979e36971defd9e10d88a56b50a3dd8b26c5 Jul 25 00:04:34.680177 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:34.680132 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" event={"ID":"54aef27f-be55-456d-a798-62a090ada7ca","Type":"ContainerStarted","Data":"57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff"} Jul 25 00:04:34.680709 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:34.680248 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" event={"ID":"54aef27f-be55-456d-a798-62a090ada7ca","Type":"ContainerStarted","Data":"d2ddf67a316c000bc7f589bc30f1979e36971defd9e10d88a56b50a3dd8b26c5"} Jul 25 00:04:34.696458 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:34.696404 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" podStartSLOduration=1.6963897129999999 podStartE2EDuration="1.696389713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:04:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:04:34.694058048 +0000 UTC m=+2134.386898893" watchObservedRunningTime="2026-07-25 00:04:34.696389713 +0000 UTC m=+2134.389230540" Jul 25 00:04:35.685815 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:35.685788 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:37.781871 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.781831 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb"] Jul 25 00:04:37.786129 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.786103 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.788777 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.788753 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-107131-openshift-default-dockercfg-5plkl\"" Jul 25 00:04:37.795756 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.795471 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb"] Jul 25 00:04:37.889475 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889435 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-certs\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889693 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889492 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5499z\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-kube-api-access-5499z\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889693 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889604 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-envoy\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889818 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889691 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf4a657-ed24-4238-abed-8e940e98c28a-istiod-ca-cert\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889818 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889723 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-socket\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889818 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889782 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-credential-socket\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889971 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889836 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-podinfo\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889971 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889882 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-data\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.889971 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.889907 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-token\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.990881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.990842 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-data\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.990881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.990880 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-token\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991108 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.990934 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-certs\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991108 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991100 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5499z\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-kube-api-access-5499z\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991177 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991153 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-envoy\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991284 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991262 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf4a657-ed24-4238-abed-8e940e98c28a-istiod-ca-cert\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991448 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991386 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-socket\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991448 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991393 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-certs\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991448 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991318 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-data\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991466 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-credential-socket\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991494 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-podinfo\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991693 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-socket\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991695 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-credential-socket\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.991823 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.991812 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf4a657-ed24-4238-abed-8e940e98c28a-istiod-ca-cert\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.993867 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.993845 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-envoy\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:37.993970 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:37.993956 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-podinfo\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:38.002824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.002791 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5499z\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-kube-api-access-5499z\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:38.002937 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.002853 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-token\") pod \"e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:38.099997 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.099908 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:38.246679 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.246648 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb"] Jul 25 00:04:38.248114 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:04:38.248062 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf4a657_ed24_4238_abed_8e940e98c28a.slice/crio-dd51b61f480d7903cf8db343b163a2850a5ebfd98e2116ed8cf19251550600e6 WatchSource:0}: Error finding container dd51b61f480d7903cf8db343b163a2850a5ebfd98e2116ed8cf19251550600e6: Status 404 returned error can't find the container with id dd51b61f480d7903cf8db343b163a2850a5ebfd98e2116ed8cf19251550600e6 Jul 25 00:04:38.250489 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.250443 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:38.250575 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.250560 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:38.250613 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.250598 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:38.703306 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.703262 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" event={"ID":"dcf4a657-ed24-4238-abed-8e940e98c28a","Type":"ContainerStarted","Data":"55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17"} Jul 25 00:04:38.703491 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.703318 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" event={"ID":"dcf4a657-ed24-4238-abed-8e940e98c28a","Type":"ContainerStarted","Data":"dd51b61f480d7903cf8db343b163a2850a5ebfd98e2116ed8cf19251550600e6"} Jul 25 00:04:38.722564 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:38.722513 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podStartSLOduration=1.722497043 podStartE2EDuration="1.722497043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:04:38.721045422 +0000 UTC m=+2138.413886251" watchObservedRunningTime="2026-07-25 00:04:38.722497043 +0000 UTC m=+2138.415337872" Jul 25 00:04:39.100372 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:39.100283 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:39.105278 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:39.105226 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:39.708757 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:39.708726 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:39.709877 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:39.709853 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:41.377615 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.377584 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95"] Jul 25 00:04:41.381094 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.381076 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.383446 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.383428 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-107131-dockercfg-4glps\"" Jul 25 00:04:41.383446 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.383443 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-107131\"" Jul 25 00:04:41.391389 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.391360 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95"] Jul 25 00:04:41.411606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.411578 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8"] Jul 25 00:04:41.414914 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.414895 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.421767 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.421664 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ls9m\" (UniqueName: \"kubernetes.io/projected/751295dc-0049-4971-88b6-168f9ad60511-kube-api-access-7ls9m\") pod \"payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.421922 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.421883 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751295dc-0049-4971-88b6-168f9ad60511-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.425176 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.425154 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8"] Jul 25 00:04:41.523023 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.522987 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ls9m\" (UniqueName: \"kubernetes.io/projected/751295dc-0049-4971-88b6-168f9ad60511-kube-api-access-7ls9m\") pod \"payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.523203 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.523057 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751295dc-0049-4971-88b6-168f9ad60511-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.523203 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.523083 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476fb848-bff0-4c6c-b7c3-6a506a092c74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.523203 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.523106 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbh6\" (UniqueName: \"kubernetes.io/projected/476fb848-bff0-4c6c-b7c3-6a506a092c74-kube-api-access-hgbh6\") pod \"payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.523720 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.523702 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751295dc-0049-4971-88b6-168f9ad60511-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.530540 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.530517 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ls9m\" (UniqueName: \"kubernetes.io/projected/751295dc-0049-4971-88b6-168f9ad60511-kube-api-access-7ls9m\") pod \"payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.623827 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.623786 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476fb848-bff0-4c6c-b7c3-6a506a092c74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.623827 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.623834 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbh6\" (UniqueName: \"kubernetes.io/projected/476fb848-bff0-4c6c-b7c3-6a506a092c74-kube-api-access-hgbh6\") pod \"payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.624595 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.624574 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476fb848-bff0-4c6c-b7c3-6a506a092c74-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.631343 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.631285 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbh6\" (UniqueName: \"kubernetes.io/projected/476fb848-bff0-4c6c-b7c3-6a506a092c74-kube-api-access-hgbh6\") pod \"payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:41.694255 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.694197 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:41.726002 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:41.725977 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:42.118606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.118579 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95"] Jul 25 00:04:42.120051 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:04:42.120015 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751295dc_0049_4971_88b6_168f9ad60511.slice/crio-deba98ec8f84d52b16f8a797e1d4cb1e8b59ab0518cc5623796165c1d912c2a1 WatchSource:0}: Error finding container deba98ec8f84d52b16f8a797e1d4cb1e8b59ab0518cc5623796165c1d912c2a1: Status 404 returned error can't find the container with id deba98ec8f84d52b16f8a797e1d4cb1e8b59ab0518cc5623796165c1d912c2a1 Jul 25 00:04:42.147984 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.147948 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8"] Jul 25 00:04:42.168536 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:04:42.168505 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476fb848_bff0_4c6c_b7c3_6a506a092c74.slice/crio-c18fd440e5bda399d1fd8b009ccc1de5b22ef275fe60d2788d7867969e949d2c WatchSource:0}: Error finding container c18fd440e5bda399d1fd8b009ccc1de5b22ef275fe60d2788d7867969e949d2c: Status 404 returned error can't find the container with id c18fd440e5bda399d1fd8b009ccc1de5b22ef275fe60d2788d7867969e949d2c Jul 25 00:04:42.725610 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.725569 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" event={"ID":"476fb848-bff0-4c6c-b7c3-6a506a092c74","Type":"ContainerStarted","Data":"db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330"} Jul 25 00:04:42.726103 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.725636 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" event={"ID":"476fb848-bff0-4c6c-b7c3-6a506a092c74","Type":"ContainerStarted","Data":"c18fd440e5bda399d1fd8b009ccc1de5b22ef275fe60d2788d7867969e949d2c"} Jul 25 00:04:42.726103 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.725696 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:42.727072 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.727051 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" event={"ID":"751295dc-0049-4971-88b6-168f9ad60511","Type":"ContainerStarted","Data":"9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60"} Jul 25 00:04:42.727169 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.727090 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:42.727169 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.727107 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" event={"ID":"751295dc-0049-4971-88b6-168f9ad60511","Type":"ContainerStarted","Data":"deba98ec8f84d52b16f8a797e1d4cb1e8b59ab0518cc5623796165c1d912c2a1"} Jul 25 00:04:42.759991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.759917 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" podStartSLOduration=1.759896471 podStartE2EDuration="1.759896471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:04:42.758269619 +0000 UTC m=+2142.451110446" watchObservedRunningTime="2026-07-25 00:04:42.759896471 +0000 UTC m=+2142.452737298" Jul 25 00:04:42.761831 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:42.761778 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" podStartSLOduration=1.7617617509999999 podStartE2EDuration="1.761761751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:04:42.743512353 +0000 UTC m=+2142.436353180" watchObservedRunningTime="2026-07-25 00:04:42.761761751 +0000 UTC m=+2142.454602580" Jul 25 00:04:46.694500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:46.694465 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:04:47.671595 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.671557 2586 patch_prober.go:28] interesting pod/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 25 00:04:47.671762 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.671610 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 25 00:04:47.769895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.769857 2586 generic.go:356] "Generic (PLEG): container finished" podID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerID="215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff" exitCode=0 Jul 25 00:04:47.770276 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.769927 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" event={"ID":"ee035886-6ae0-46b9-a5c2-5d3fbcad264a","Type":"ContainerDied","Data":"215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff"} Jul 25 00:04:47.770337 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.770324 2586 scope.go:117] "RemoveContainer" containerID="215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff" Jul 25 00:04:47.771361 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.771336 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:47.771428 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.771401 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:47.771468 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:47.771433 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:48.781050 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:48.781014 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" event={"ID":"ee035886-6ae0-46b9-a5c2-5d3fbcad264a","Type":"ContainerStarted","Data":"684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4"} Jul 25 00:04:49.728916 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:49.728878 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:49.729132 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:49.729108 2586 patch_prober.go:28] interesting pod/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 25 00:04:49.729190 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:49.729158 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 25 00:04:50.728887 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:50.728848 2586 patch_prober.go:28] interesting pod/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 25 00:04:50.729312 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:50.728906 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 25 00:04:51.728578 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:51.728537 2586 patch_prober.go:28] interesting pod/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 25 00:04:51.728811 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:51.728602 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 25 00:04:52.728482 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:52.728444 2586 patch_prober.go:28] interesting pod/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 25 00:04:52.728886 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:52.728499 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 25 00:04:53.733881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:53.733800 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:04:53.734224 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:53.733995 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:04:53.734224 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:53.734104 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:53.735071 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:53.735049 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:53.735814 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:53.735794 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:04:54.709966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.709928 2586 patch_prober.go:28] interesting pod/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 25 00:04:54.710185 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.709988 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 25 00:04:54.820148 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.820115 2586 generic.go:356] "Generic (PLEG): container finished" podID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerID="55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17" exitCode=0 Jul 25 00:04:54.820582 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.820196 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" event={"ID":"dcf4a657-ed24-4238-abed-8e940e98c28a","Type":"ContainerDied","Data":"55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17"} Jul 25 00:04:54.820715 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.820701 2586 scope.go:117] "RemoveContainer" containerID="55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17" Jul 25 00:04:54.821705 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.821675 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:54.821829 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.821741 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:54.821829 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:54.821773 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:04:55.828305 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:55.828262 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" event={"ID":"dcf4a657-ed24-4238-abed-8e940e98c28a","Type":"ContainerStarted","Data":"1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306"} Jul 25 00:04:56.100184 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:56.100061 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:04:56.100383 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:56.100309 2586 patch_prober.go:28] interesting pod/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 25 00:04:56.100457 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:56.100374 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 25 00:04:57.101302 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:57.101218 2586 patch_prober.go:28] interesting pod/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 25 00:04:57.101717 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:57.101324 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 25 00:04:58.101059 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:58.101020 2586 patch_prober.go:28] interesting pod/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 25 00:04:58.101286 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:58.101091 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 25 00:04:59.100524 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:59.100489 2586 patch_prober.go:28] interesting pod/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 25 00:04:59.100931 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:04:59.100547 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 25 00:05:00.104407 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:05:00.104369 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:05:00.104798 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:05:00.104638 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:05:00.105310 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:05:00.105289 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:05:01.055883 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:05:01.055852 2586 scope.go:117] "RemoveContainer" containerID="42f6fbca2c7b10e0c1771c50e69c9eb90c9e97b2be895d2e88ac83f29e8b990a" Jul 25 00:06:16.563349 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.563312 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92"] Jul 25 00:06:16.563918 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.563568 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" podUID="54aef27f-be55-456d-a798-62a090ada7ca" containerName="payload-processing" containerID="cri-o://57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff" gracePeriod=30 Jul 25 00:06:16.692471 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.692428 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" podUID="54aef27f-be55-456d-a798-62a090ada7ca" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.66:9004: connect: connection refused" Jul 25 00:06:16.839862 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.839831 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:06:16.900491 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.900454 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5w6vg\" (UniqueName: \"kubernetes.io/projected/54aef27f-be55-456d-a798-62a090ada7ca-kube-api-access-5w6vg\") pod \"54aef27f-be55-456d-a798-62a090ada7ca\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " Jul 25 00:06:16.900690 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.900513 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54aef27f-be55-456d-a798-62a090ada7ca-plugins-config-volume\") pod \"54aef27f-be55-456d-a798-62a090ada7ca\" (UID: \"54aef27f-be55-456d-a798-62a090ada7ca\") " Jul 25 00:06:16.900896 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.900871 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54aef27f-be55-456d-a798-62a090ada7ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "54aef27f-be55-456d-a798-62a090ada7ca" (UID: "54aef27f-be55-456d-a798-62a090ada7ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:06:16.902697 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:16.902674 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54aef27f-be55-456d-a798-62a090ada7ca-kube-api-access-5w6vg" (OuterVolumeSpecName: "kube-api-access-5w6vg") pod "54aef27f-be55-456d-a798-62a090ada7ca" (UID: "54aef27f-be55-456d-a798-62a090ada7ca"). InnerVolumeSpecName "kube-api-access-5w6vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:17.001736 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.001697 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w6vg\" (UniqueName: \"kubernetes.io/projected/54aef27f-be55-456d-a798-62a090ada7ca-kube-api-access-5w6vg\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:17.001736 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.001729 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54aef27f-be55-456d-a798-62a090ada7ca-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:17.268928 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.268885 2586 generic.go:356] "Generic (PLEG): container finished" podID="54aef27f-be55-456d-a798-62a090ada7ca" containerID="57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff" exitCode=0 Jul 25 00:06:17.268928 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.268929 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" Jul 25 00:06:17.269149 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.268971 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" event={"ID":"54aef27f-be55-456d-a798-62a090ada7ca","Type":"ContainerDied","Data":"57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff"} Jul 25 00:06:17.269149 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.269029 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92" event={"ID":"54aef27f-be55-456d-a798-62a090ada7ca","Type":"ContainerDied","Data":"d2ddf67a316c000bc7f589bc30f1979e36971defd9e10d88a56b50a3dd8b26c5"} Jul 25 00:06:17.269149 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.269061 2586 scope.go:117] "RemoveContainer" containerID="57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff" Jul 25 00:06:17.279380 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.279357 2586 scope.go:117] "RemoveContainer" containerID="57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff" Jul 25 00:06:17.279680 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:17.279660 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff\": container with ID starting with 57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff not found: ID does not exist" containerID="57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff" Jul 25 00:06:17.279726 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.279692 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff"} err="failed to get container status \"57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff\": rpc error: code = NotFound desc = could not find container \"57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff\": container with ID starting with 57b2af43a3e8c5eab4cb6b42a82dfe23b8bad7e1abd260b68e245a8f736643ff not found: ID does not exist" Jul 25 00:06:17.291149 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.291114 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92"] Jul 25 00:06:17.295976 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:17.295943 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-820f12-cf68794d8-62r92"] Jul 25 00:06:18.787840 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:18.787805 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54aef27f-be55-456d-a798-62a090ada7ca" path="/var/lib/kubelet/pods/54aef27f-be55-456d-a798-62a090ada7ca/volumes" Jul 25 00:06:27.596908 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:27.596821 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq"] Jul 25 00:06:27.597351 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:27.597053 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" containerID="cri-o://684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4" gracePeriod=30 Jul 25 00:06:32.868186 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.868104 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:06:32.947303 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947268 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7plf\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-kube-api-access-d7plf\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947320 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-podinfo\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947345 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-token\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947388 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-data\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947430 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-credential-socket\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947460 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-socket\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947491 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istiod-ca-cert\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947812 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947570 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-certs\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947812 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947618 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-envoy\") pod \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\" (UID: \"ee035886-6ae0-46b9-a5c2-5d3fbcad264a\") " Jul 25 00:06:32.947939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947798 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:32.947939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947906 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:06:32.948067 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.947964 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:32.948125 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.948076 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:32.948295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.948270 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:32.948554 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.948530 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:32.949831 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.949785 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:06:32.949924 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.949866 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:32.949924 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.949884 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-kube-api-access-d7plf" (OuterVolumeSpecName: "kube-api-access-d7plf") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "kube-api-access-d7plf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:32.950011 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:32.949939 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee035886-6ae0-46b9-a5c2-5d3fbcad264a" (UID: "ee035886-6ae0-46b9-a5c2-5d3fbcad264a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:33.048608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048562 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048600 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7plf\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-kube-api-access-d7plf\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048612 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048608 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048621 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048629 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048636 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048645 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.048882 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.048653 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee035886-6ae0-46b9-a5c2-5d3fbcad264a-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:33.358784 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.358751 2586 generic.go:356] "Generic (PLEG): container finished" podID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerID="684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4" exitCode=0 Jul 25 00:06:33.358936 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.358800 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" Jul 25 00:06:33.358936 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.358836 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" event={"ID":"ee035886-6ae0-46b9-a5c2-5d3fbcad264a","Type":"ContainerDied","Data":"684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4"} Jul 25 00:06:33.358936 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.358893 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq" event={"ID":"ee035886-6ae0-46b9-a5c2-5d3fbcad264a","Type":"ContainerDied","Data":"cccd07aa4f1968bdac7d17db0a93dacd58430209cd80cded5837f34ec32ee177"} Jul 25 00:06:33.358936 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.358917 2586 scope.go:117] "RemoveContainer" containerID="684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4" Jul 25 00:06:33.368642 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.368621 2586 scope.go:117] "RemoveContainer" containerID="215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff" Jul 25 00:06:33.376841 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.376820 2586 scope.go:117] "RemoveContainer" containerID="684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4" Jul 25 00:06:33.377130 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:33.377104 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4\": container with ID starting with 684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4 not found: ID does not exist" containerID="684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4" Jul 25 00:06:33.377130 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.377137 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4"} err="failed to get container status \"684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4\": rpc error: code = NotFound desc = could not find container \"684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4\": container with ID starting with 684769217bde9cd7afb79e8f423d8100bf2e6a63883e2651d2c24ae415378bb4 not found: ID does not exist" Jul 25 00:06:33.377130 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.377162 2586 scope.go:117] "RemoveContainer" containerID="215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff" Jul 25 00:06:33.377656 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:33.377621 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff\": container with ID starting with 215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff not found: ID does not exist" containerID="215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff" Jul 25 00:06:33.377755 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.377661 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff"} err="failed to get container status \"215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff\": rpc error: code = NotFound desc = could not find container \"215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff\": container with ID starting with 215e1a279ff468469ddeef3d84e7606f7bd0a38c5462644c9b02da7e2d895fff not found: ID does not exist" Jul 25 00:06:33.380055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.380034 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq"] Jul 25 00:06:33.383107 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:33.383085 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-820f12-openshift-default-7f767497bd-pq6bq"] Jul 25 00:06:34.786816 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:34.786782 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" path="/var/lib/kubelet/pods/ee035886-6ae0-46b9-a5c2-5d3fbcad264a/volumes" Jul 25 00:06:38.436701 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.436660 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8"] Jul 25 00:06:38.437224 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.436924 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" podUID="476fb848-bff0-4c6c-b7c3-6a506a092c74" containerName="payload-processing" containerID="cri-o://db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330" gracePeriod=30 Jul 25 00:06:38.444965 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.444930 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95"] Jul 25 00:06:38.445199 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.445167 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" podUID="751295dc-0049-4971-88b6-168f9ad60511" containerName="payload-pre-processing" containerID="cri-o://9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60" gracePeriod=30 Jul 25 00:06:38.732439 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.732412 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:06:38.738835 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.738795 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:06:38.801904 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.801871 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751295dc-0049-4971-88b6-168f9ad60511-plugins-config-volume\") pod \"751295dc-0049-4971-88b6-168f9ad60511\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " Jul 25 00:06:38.802096 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.801918 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7ls9m\" (UniqueName: \"kubernetes.io/projected/751295dc-0049-4971-88b6-168f9ad60511-kube-api-access-7ls9m\") pod \"751295dc-0049-4971-88b6-168f9ad60511\" (UID: \"751295dc-0049-4971-88b6-168f9ad60511\") " Jul 25 00:06:38.802096 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.801937 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hgbh6\" (UniqueName: \"kubernetes.io/projected/476fb848-bff0-4c6c-b7c3-6a506a092c74-kube-api-access-hgbh6\") pod \"476fb848-bff0-4c6c-b7c3-6a506a092c74\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " Jul 25 00:06:38.802096 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.801955 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476fb848-bff0-4c6c-b7c3-6a506a092c74-plugins-config-volume\") pod \"476fb848-bff0-4c6c-b7c3-6a506a092c74\" (UID: \"476fb848-bff0-4c6c-b7c3-6a506a092c74\") " Jul 25 00:06:38.802364 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.802334 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751295dc-0049-4971-88b6-168f9ad60511-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "751295dc-0049-4971-88b6-168f9ad60511" (UID: "751295dc-0049-4971-88b6-168f9ad60511"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:06:38.802419 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.802346 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476fb848-bff0-4c6c-b7c3-6a506a092c74-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "476fb848-bff0-4c6c-b7c3-6a506a092c74" (UID: "476fb848-bff0-4c6c-b7c3-6a506a092c74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:06:38.804073 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.804040 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476fb848-bff0-4c6c-b7c3-6a506a092c74-kube-api-access-hgbh6" (OuterVolumeSpecName: "kube-api-access-hgbh6") pod "476fb848-bff0-4c6c-b7c3-6a506a092c74" (UID: "476fb848-bff0-4c6c-b7c3-6a506a092c74"). InnerVolumeSpecName "kube-api-access-hgbh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:38.804174 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.804126 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751295dc-0049-4971-88b6-168f9ad60511-kube-api-access-7ls9m" (OuterVolumeSpecName: "kube-api-access-7ls9m") pod "751295dc-0049-4971-88b6-168f9ad60511" (UID: "751295dc-0049-4971-88b6-168f9ad60511"). InnerVolumeSpecName "kube-api-access-7ls9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:38.903121 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.903076 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/751295dc-0049-4971-88b6-168f9ad60511-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:38.903121 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.903117 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ls9m\" (UniqueName: \"kubernetes.io/projected/751295dc-0049-4971-88b6-168f9ad60511-kube-api-access-7ls9m\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:38.903371 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.903133 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbh6\" (UniqueName: \"kubernetes.io/projected/476fb848-bff0-4c6c-b7c3-6a506a092c74-kube-api-access-hgbh6\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:38.903371 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:38.903148 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/476fb848-bff0-4c6c-b7c3-6a506a092c74-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:39.396258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.396196 2586 generic.go:356] "Generic (PLEG): container finished" podID="476fb848-bff0-4c6c-b7c3-6a506a092c74" containerID="db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330" exitCode=0 Jul 25 00:06:39.396258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.396263 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" Jul 25 00:06:39.396488 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.396291 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" event={"ID":"476fb848-bff0-4c6c-b7c3-6a506a092c74","Type":"ContainerDied","Data":"db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330"} Jul 25 00:06:39.396488 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.396352 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8" event={"ID":"476fb848-bff0-4c6c-b7c3-6a506a092c74","Type":"ContainerDied","Data":"c18fd440e5bda399d1fd8b009ccc1de5b22ef275fe60d2788d7867969e949d2c"} Jul 25 00:06:39.396488 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.396379 2586 scope.go:117] "RemoveContainer" containerID="db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330" Jul 25 00:06:39.397633 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.397611 2586 generic.go:356] "Generic (PLEG): container finished" podID="751295dc-0049-4971-88b6-168f9ad60511" containerID="9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60" exitCode=0 Jul 25 00:06:39.397737 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.397658 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" Jul 25 00:06:39.397737 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.397684 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" event={"ID":"751295dc-0049-4971-88b6-168f9ad60511","Type":"ContainerDied","Data":"9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60"} Jul 25 00:06:39.397737 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.397715 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95" event={"ID":"751295dc-0049-4971-88b6-168f9ad60511","Type":"ContainerDied","Data":"deba98ec8f84d52b16f8a797e1d4cb1e8b59ab0518cc5623796165c1d912c2a1"} Jul 25 00:06:39.405930 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.405909 2586 scope.go:117] "RemoveContainer" containerID="db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330" Jul 25 00:06:39.406213 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:39.406191 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330\": container with ID starting with db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330 not found: ID does not exist" containerID="db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330" Jul 25 00:06:39.406307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.406222 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330"} err="failed to get container status \"db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330\": rpc error: code = NotFound desc = could not find container \"db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330\": container with ID starting with db8770d9b203ce77e46ccf46eacfb72573601960414721dd78652a6830b99330 not found: ID does not exist" Jul 25 00:06:39.406307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.406260 2586 scope.go:117] "RemoveContainer" containerID="9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60" Jul 25 00:06:39.414607 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.414565 2586 scope.go:117] "RemoveContainer" containerID="9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60" Jul 25 00:06:39.414854 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:39.414833 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60\": container with ID starting with 9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60 not found: ID does not exist" containerID="9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60" Jul 25 00:06:39.414912 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.414864 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60"} err="failed to get container status \"9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60\": rpc error: code = NotFound desc = could not find container \"9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60\": container with ID starting with 9804789494bdfd6644e8047c50774a06bfd90e35feedbe75f313223970e38e60 not found: ID does not exist" Jul 25 00:06:39.419619 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.419594 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95"] Jul 25 00:06:39.421864 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.421844 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-107131-c947d85bd-6qg95"] Jul 25 00:06:39.430748 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.430728 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8"] Jul 25 00:06:39.434040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:39.434018 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-107131-6bbbcdb6fd-ndbc8"] Jul 25 00:06:40.793824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:40.793787 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="476fb848-bff0-4c6c-b7c3-6a506a092c74" path="/var/lib/kubelet/pods/476fb848-bff0-4c6c-b7c3-6a506a092c74/volumes" Jul 25 00:06:40.794484 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:40.794456 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751295dc-0049-4971-88b6-168f9ad60511" path="/var/lib/kubelet/pods/751295dc-0049-4971-88b6-168f9ad60511/volumes" Jul 25 00:06:49.619312 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:49.619273 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb"] Jul 25 00:06:49.619872 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:49.619507 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" containerID="cri-o://1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306" gracePeriod=30 Jul 25 00:06:50.524325 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524286 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k"] Jul 25 00:06:50.524881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524862 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" Jul 25 00:06:50.524881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524882 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524896 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54aef27f-be55-456d-a798-62a090ada7ca" containerName="payload-processing" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524903 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="54aef27f-be55-456d-a798-62a090ada7ca" containerName="payload-processing" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524914 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="476fb848-bff0-4c6c-b7c3-6a506a092c74" containerName="payload-processing" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524920 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="476fb848-bff0-4c6c-b7c3-6a506a092c74" containerName="payload-processing" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524932 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524938 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524950 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751295dc-0049-4971-88b6-168f9ad60511" containerName="payload-pre-processing" Jul 25 00:06:50.525008 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.524956 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="751295dc-0049-4971-88b6-168f9ad60511" containerName="payload-pre-processing" Jul 25 00:06:50.525424 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.525014 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="54aef27f-be55-456d-a798-62a090ada7ca" containerName="payload-processing" Jul 25 00:06:50.525424 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.525022 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="751295dc-0049-4971-88b6-168f9ad60511" containerName="payload-pre-processing" Jul 25 00:06:50.525424 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.525028 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" Jul 25 00:06:50.525424 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.525037 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="476fb848-bff0-4c6c-b7c3-6a506a092c74" containerName="payload-processing" Jul 25 00:06:50.528367 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.528339 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.530704 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.530681 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-97aaee-openshift-default-dockercfg-5b46q\"" Jul 25 00:06:50.538988 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.538963 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k"] Jul 25 00:06:50.616613 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616561 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istiod-ca-cert\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.616834 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616641 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-socket\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.616834 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616666 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-envoy\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.616834 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616694 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-certs\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.616834 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616801 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-token\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.617016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616866 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-podinfo\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.617016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616888 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-credential-socket\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.617016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616908 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhlfm\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-kube-api-access-xhlfm\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.617016 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.616932 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-data\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718436 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718398 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-token\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718457 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-podinfo\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718485 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-credential-socket\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718506 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhlfm\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-kube-api-access-xhlfm\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718528 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-data\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718576 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istiod-ca-cert\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718618 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-socket\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718645 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-envoy\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.718919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718671 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-certs\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.719372 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.718956 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-credential-socket\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.719372 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.719083 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-data\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.719372 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.719146 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-socket\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.719372 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.719315 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-certs\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.719534 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.719465 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istiod-ca-cert\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.721061 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.721038 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-envoy\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.721163 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.721148 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-podinfo\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.725855 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.725825 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-token\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.725982 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.725960 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhlfm\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-kube-api-access-xhlfm\") pod \"e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.842826 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.842731 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:50.994108 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.994080 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k"] Jul 25 00:06:50.996442 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:06:50.996402 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7e702f_5e99_4ea6_844f_4b8c19eaebeb.slice/crio-aa4454680cb5b70e77e5b025feaafd7ecfc81def3d7e0003bd731c11954a073c WatchSource:0}: Error finding container aa4454680cb5b70e77e5b025feaafd7ecfc81def3d7e0003bd731c11954a073c: Status 404 returned error can't find the container with id aa4454680cb5b70e77e5b025feaafd7ecfc81def3d7e0003bd731c11954a073c Jul 25 00:06:50.998782 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.998745 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:06:50.998893 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.998829 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:06:50.998893 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:50.998878 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:06:51.461704 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:51.461669 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" event={"ID":"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb","Type":"ContainerStarted","Data":"937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821"} Jul 25 00:06:51.461907 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:51.461738 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" event={"ID":"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb","Type":"ContainerStarted","Data":"aa4454680cb5b70e77e5b025feaafd7ecfc81def3d7e0003bd731c11954a073c"} Jul 25 00:06:51.479551 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:51.479505 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" podStartSLOduration=1.479488032 podStartE2EDuration="1.479488032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:06:51.479042587 +0000 UTC m=+2271.171883425" watchObservedRunningTime="2026-07-25 00:06:51.479488032 +0000 UTC m=+2271.172328861" Jul 25 00:06:51.843755 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:51.843672 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:52.848728 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:52.848698 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:53.471771 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:53.471741 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:53.472769 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:53.472748 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:06:54.900642 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:54.900614 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:06:55.064455 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064417 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5499z\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-kube-api-access-5499z\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064526 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-envoy\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064599 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-credential-socket\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064631 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-certs\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064652 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-socket\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064873 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064677 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-data\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064873 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064706 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-token\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.064873 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064735 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-podinfo\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.065025 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.064970 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:55.065025 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065005 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:55.065139 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065022 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:55.065139 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065074 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf4a657-ed24-4238-abed-8e940e98c28a-istiod-ca-cert\") pod \"dcf4a657-ed24-4238-abed-8e940e98c28a\" (UID: \"dcf4a657-ed24-4238-abed-8e940e98c28a\") " Jul 25 00:06:55.065517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065444 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf4a657-ed24-4238-abed-8e940e98c28a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:06:55.065517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065454 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.065517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065500 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.065517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065518 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.065816 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.065553 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-data" (OuterVolumeSpecName: "istio-data") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:55.066894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.066866 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:06:55.066894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.066876 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-token" (OuterVolumeSpecName: "istio-token") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:55.067030 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.066935 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-kube-api-access-5499z" (OuterVolumeSpecName: "kube-api-access-5499z") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "kube-api-access-5499z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:06:55.067030 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.066957 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dcf4a657-ed24-4238-abed-8e940e98c28a" (UID: "dcf4a657-ed24-4238-abed-8e940e98c28a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:06:55.166734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.166646 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.166734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.166677 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.166734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.166685 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.166734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.166695 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf4a657-ed24-4238-abed-8e940e98c28a-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.166734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.166703 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf4a657-ed24-4238-abed-8e940e98c28a-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.166734 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.166712 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5499z\" (UniqueName: \"kubernetes.io/projected/dcf4a657-ed24-4238-abed-8e940e98c28a-kube-api-access-5499z\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:06:55.484953 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.484858 2586 generic.go:356] "Generic (PLEG): container finished" podID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerID="1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306" exitCode=0 Jul 25 00:06:55.484953 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.484907 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" Jul 25 00:06:55.485190 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.484948 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" event={"ID":"dcf4a657-ed24-4238-abed-8e940e98c28a","Type":"ContainerDied","Data":"1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306"} Jul 25 00:06:55.485190 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.485010 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb" event={"ID":"dcf4a657-ed24-4238-abed-8e940e98c28a","Type":"ContainerDied","Data":"dd51b61f480d7903cf8db343b163a2850a5ebfd98e2116ed8cf19251550600e6"} Jul 25 00:06:55.485190 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.485036 2586 scope.go:117] "RemoveContainer" containerID="1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306" Jul 25 00:06:55.495527 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.495508 2586 scope.go:117] "RemoveContainer" containerID="55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17" Jul 25 00:06:55.504358 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.504316 2586 scope.go:117] "RemoveContainer" containerID="1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306" Jul 25 00:06:55.504659 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:55.504634 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306\": container with ID starting with 1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306 not found: ID does not exist" containerID="1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306" Jul 25 00:06:55.504784 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.504671 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306"} err="failed to get container status \"1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306\": rpc error: code = NotFound desc = could not find container \"1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306\": container with ID starting with 1723b13a9589b152855013978eeaf1cbb712cef6e5c9f07aaecf40683c9c4306 not found: ID does not exist" Jul 25 00:06:55.504784 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.504696 2586 scope.go:117] "RemoveContainer" containerID="55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17" Jul 25 00:06:55.505047 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:06:55.505026 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17\": container with ID starting with 55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17 not found: ID does not exist" containerID="55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17" Jul 25 00:06:55.505157 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.505056 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17"} err="failed to get container status \"55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17\": rpc error: code = NotFound desc = could not find container \"55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17\": container with ID starting with 55462c131a3a005ae7f1d036c44f642f81feda091bb744593b04a68687c19f17 not found: ID does not exist" Jul 25 00:06:55.506395 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.506376 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb"] Jul 25 00:06:55.510334 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:55.510305 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-107131-openshift-default-fcb7fc578-6rdhb"] Jul 25 00:06:56.790526 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:56.790487 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" path="/var/lib/kubelet/pods/dcf4a657-ed24-4238-abed-8e940e98c28a/volumes" Jul 25 00:06:58.749773 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.749736 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94"] Jul 25 00:06:58.750268 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.750155 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" Jul 25 00:06:58.750268 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.750168 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" Jul 25 00:06:58.750404 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.750280 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" Jul 25 00:06:58.750404 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.750294 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee035886-6ae0-46b9-a5c2-5d3fbcad264a" containerName="istio-proxy" Jul 25 00:06:58.753776 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.753753 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.756542 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.756520 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-7d3cec-openshift-default-dockercfg-tz856\"" Jul 25 00:06:58.765022 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.764994 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94"] Jul 25 00:06:58.902792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.902748 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-certs\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.902792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.902793 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-socket\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.902861 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-podinfo\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.902913 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-token\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.902971 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-envoy\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.903011 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-data\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.903082 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7664940-a25f-4fdf-9be9-e0ef71e85426-istiod-ca-cert\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.903144 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-credential-socket\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:58.903300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:58.903177 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhlb\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-kube-api-access-wkhlb\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.004722 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004624 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-envoy\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.004722 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004663 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-data\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.004722 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004685 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7664940-a25f-4fdf-9be9-e0ef71e85426-istiod-ca-cert\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004738 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-credential-socket\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004768 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhlb\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-kube-api-access-wkhlb\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004798 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-certs\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004820 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-socket\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004855 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-podinfo\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.004945 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-token\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005318 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.005216 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-socket\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005448 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.005424 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-data\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005448 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.005438 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-credential-socket\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005562 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.005536 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7664940-a25f-4fdf-9be9-e0ef71e85426-istiod-ca-cert\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.005675 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.005653 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-certs\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.007177 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.007155 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-envoy\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.007431 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.007411 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-podinfo\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.012690 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.012672 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-token\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.012865 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.012846 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhlb\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-kube-api-access-wkhlb\") pod \"e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.067458 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.067416 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:06:59.265403 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.263808 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94"] Jul 25 00:06:59.267808 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.267673 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:06:59.267808 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.267762 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:06:59.267808 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.267805 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:06:59.511742 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.511639 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" event={"ID":"a7664940-a25f-4fdf-9be9-e0ef71e85426","Type":"ContainerStarted","Data":"de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8"} Jul 25 00:06:59.511742 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.511708 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" event={"ID":"a7664940-a25f-4fdf-9be9-e0ef71e85426","Type":"ContainerStarted","Data":"022d1ef08fd8737ca5ae2ae39faf97028fa21eda8cea1644047fb0ec6e39e946"} Jul 25 00:06:59.532966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:06:59.532904 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podStartSLOduration=1.532885917 podStartE2EDuration="1.532885917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:06:59.531057997 +0000 UTC m=+2279.223898826" watchObservedRunningTime="2026-07-25 00:06:59.532885917 +0000 UTC m=+2279.225726744" Jul 25 00:07:00.067814 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:00.067770 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:00.073098 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:00.073069 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:00.517015 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:00.516985 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:00.517925 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:00.517907 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:02.428040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.427987 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns"] Jul 25 00:07:02.428495 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.428478 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" Jul 25 00:07:02.428561 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.428498 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" Jul 25 00:07:02.428595 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.428577 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcf4a657-ed24-4238-abed-8e940e98c28a" containerName="istio-proxy" Jul 25 00:07:02.431628 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.431603 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.433001 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.432981 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e1dc38-569d-47ee-aafa-c9ab4a993682-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.433164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.433143 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfvc\" (UniqueName: \"kubernetes.io/projected/e1e1dc38-569d-47ee-aafa-c9ab4a993682-kube-api-access-2lfvc\") pod \"payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.434012 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.433990 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7d3cec\"" Jul 25 00:07:02.450850 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.450735 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns"] Jul 25 00:07:02.477980 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.477940 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2"] Jul 25 00:07:02.481017 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.480994 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.483804 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.483657 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7d3cec-dockercfg-46nl9\"" Jul 25 00:07:02.483961 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.483867 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2"] Jul 25 00:07:02.533841 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.533806 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpdd\" (UniqueName: \"kubernetes.io/projected/00354262-90e1-4e1d-a7bf-e0b3e807dff9-kube-api-access-jjpdd\") pod \"payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.533986 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.533866 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00354262-90e1-4e1d-a7bf-e0b3e807dff9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.534061 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.534047 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e1dc38-569d-47ee-aafa-c9ab4a993682-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.534103 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.534069 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfvc\" (UniqueName: \"kubernetes.io/projected/e1e1dc38-569d-47ee-aafa-c9ab4a993682-kube-api-access-2lfvc\") pod \"payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.543263 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.540109 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e1dc38-569d-47ee-aafa-c9ab4a993682-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.568262 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.567997 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfvc\" (UniqueName: \"kubernetes.io/projected/e1e1dc38-569d-47ee-aafa-c9ab4a993682-kube-api-access-2lfvc\") pod \"payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.635318 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.635223 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpdd\" (UniqueName: \"kubernetes.io/projected/00354262-90e1-4e1d-a7bf-e0b3e807dff9-kube-api-access-jjpdd\") pod \"payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.635550 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.635338 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00354262-90e1-4e1d-a7bf-e0b3e807dff9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.636373 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.636341 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00354262-90e1-4e1d-a7bf-e0b3e807dff9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.650104 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.646203 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpdd\" (UniqueName: \"kubernetes.io/projected/00354262-90e1-4e1d-a7bf-e0b3e807dff9-kube-api-access-jjpdd\") pod \"payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:02.765733 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.765633 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:02.794622 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:02.794378 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:03.014036 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.014005 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2"] Jul 25 00:07:03.015157 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:07:03.015132 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00354262_90e1_4e1d_a7bf_e0b3e807dff9.slice/crio-151999ab861058c546e7c57f9fc300d019f3c874d7b9b970f059efa870f98e85 WatchSource:0}: Error finding container 151999ab861058c546e7c57f9fc300d019f3c874d7b9b970f059efa870f98e85: Status 404 returned error can't find the container with id 151999ab861058c546e7c57f9fc300d019f3c874d7b9b970f059efa870f98e85 Jul 25 00:07:03.158204 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.158142 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns"] Jul 25 00:07:03.170075 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:07:03.169380 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e1dc38_569d_47ee_aafa_c9ab4a993682.slice/crio-2949c1eccef595cb3ec386e4e6f76dc89ff3bf576e2ef5abb908b15460ac4261 WatchSource:0}: Error finding container 2949c1eccef595cb3ec386e4e6f76dc89ff3bf576e2ef5abb908b15460ac4261: Status 404 returned error can't find the container with id 2949c1eccef595cb3ec386e4e6f76dc89ff3bf576e2ef5abb908b15460ac4261 Jul 25 00:07:03.537012 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.536968 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" event={"ID":"00354262-90e1-4e1d-a7bf-e0b3e807dff9","Type":"ContainerStarted","Data":"7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917"} Jul 25 00:07:03.537501 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.537037 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" event={"ID":"00354262-90e1-4e1d-a7bf-e0b3e807dff9","Type":"ContainerStarted","Data":"151999ab861058c546e7c57f9fc300d019f3c874d7b9b970f059efa870f98e85"} Jul 25 00:07:03.537864 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.537841 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:03.539645 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.539609 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" event={"ID":"e1e1dc38-569d-47ee-aafa-c9ab4a993682","Type":"ContainerStarted","Data":"fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd"} Jul 25 00:07:03.539776 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.539660 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" event={"ID":"e1e1dc38-569d-47ee-aafa-c9ab4a993682","Type":"ContainerStarted","Data":"2949c1eccef595cb3ec386e4e6f76dc89ff3bf576e2ef5abb908b15460ac4261"} Jul 25 00:07:03.539776 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.539770 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:03.557488 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.557426 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" podStartSLOduration=1.5574087140000001 podStartE2EDuration="1.557408714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:07:03.554748257 +0000 UTC m=+2283.247589084" watchObservedRunningTime="2026-07-25 00:07:03.557408714 +0000 UTC m=+2283.250249543" Jul 25 00:07:03.571018 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:03.570959 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" podStartSLOduration=1.570939334 podStartE2EDuration="1.570939334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:07:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:07:03.569049335 +0000 UTC m=+2283.261890163" watchObservedRunningTime="2026-07-25 00:07:03.570939334 +0000 UTC m=+2283.263780167" Jul 25 00:07:06.560532 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:06.560436 2586 generic.go:356] "Generic (PLEG): container finished" podID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerID="937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821" exitCode=0 Jul 25 00:07:06.560532 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:06.560517 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" event={"ID":"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb","Type":"ContainerDied","Data":"937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821"} Jul 25 00:07:06.560964 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:06.560925 2586 scope.go:117] "RemoveContainer" containerID="937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821" Jul 25 00:07:06.562088 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:06.562061 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:07:06.562164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:06.562147 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:07:06.562201 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:06.562188 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:07:07.572406 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:07.572360 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" event={"ID":"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb","Type":"ContainerStarted","Data":"061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe"} Jul 25 00:07:07.844725 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:07.843690 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:07:07.844914 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:07.844785 2586 patch_prober.go:28] interesting pod/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 25 00:07:07.844914 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:07.844862 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 25 00:07:08.472202 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:08.472159 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:07:08.843574 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:08.843472 2586 patch_prober.go:28] interesting pod/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 25 00:07:08.843574 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:08.843539 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 25 00:07:09.843654 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:09.843611 2586 patch_prober.go:28] interesting pod/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 25 00:07:09.844310 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:09.844278 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 25 00:07:10.848262 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:10.847352 2586 patch_prober.go:28] interesting pod/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 25 00:07:10.848262 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:10.847424 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 25 00:07:11.848022 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:11.847980 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:07:11.849140 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:11.849115 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:07:14.548848 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.548812 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:07:14.618350 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.618314 2586 generic.go:356] "Generic (PLEG): container finished" podID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerID="de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8" exitCode=0 Jul 25 00:07:14.618568 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.618392 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" event={"ID":"a7664940-a25f-4fdf-9be9-e0ef71e85426","Type":"ContainerDied","Data":"de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8"} Jul 25 00:07:14.618901 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.618883 2586 scope.go:117] "RemoveContainer" containerID="de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8" Jul 25 00:07:14.620217 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.620184 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:07:14.620345 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.620308 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:07:14.620422 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:14.620345 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:07:15.517696 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:15.517643 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:15.557482 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:15.557440 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:07:15.626321 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:15.626284 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" event={"ID":"a7664940-a25f-4fdf-9be9-e0ef71e85426","Type":"ContainerStarted","Data":"cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6"} Jul 25 00:07:16.067660 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:16.067615 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:16.068926 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:16.068895 2586 patch_prober.go:28] interesting pod/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 25 00:07:16.069040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:16.068958 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 25 00:07:17.067941 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:17.067896 2586 patch_prober.go:28] interesting pod/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 25 00:07:17.068386 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:17.067961 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 25 00:07:18.068206 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:18.068154 2586 patch_prober.go:28] interesting pod/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 25 00:07:18.068600 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:18.068302 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 25 00:07:19.068122 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:19.068081 2586 patch_prober.go:28] interesting pod/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 25 00:07:19.068326 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:19.068203 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 25 00:07:20.072294 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:20.072232 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:20.072772 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:20.072751 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:20.073545 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:20.073527 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:07:50.556082 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:50.556042 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k"] Jul 25 00:07:50.556792 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:50.556342 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" containerID="cri-o://061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe" gracePeriod=30 Jul 25 00:07:55.816929 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.816869 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:07:55.854725 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854695 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-credential-socket\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854725 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854726 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istiod-ca-cert\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854745 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-socket\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854761 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-envoy\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854791 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-certs\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854840 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-token\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854881 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-podinfo\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.854983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854933 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xhlfm\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-kube-api-access-xhlfm\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.855308 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.854990 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-data\") pod \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\" (UID: \"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb\") " Jul 25 00:07:55.855308 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.855090 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:07:55.855308 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.855196 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:07:55.855480 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.855432 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.855480 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.855455 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.855594 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.855528 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:07:55.855594 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.855539 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:07:55.856050 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.856008 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:07:55.857430 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.857401 2586 generic.go:356] "Generic (PLEG): container finished" podID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerID="061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe" exitCode=0 Jul 25 00:07:55.857565 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.857476 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" Jul 25 00:07:55.857565 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.857490 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" event={"ID":"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb","Type":"ContainerDied","Data":"061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe"} Jul 25 00:07:55.857565 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.857552 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k" event={"ID":"6b7e702f-5e99-4ea6-844f-4b8c19eaebeb","Type":"ContainerDied","Data":"aa4454680cb5b70e77e5b025feaafd7ecfc81def3d7e0003bd731c11954a073c"} Jul 25 00:07:55.857725 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.857586 2586 scope.go:117] "RemoveContainer" containerID="061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe" Jul 25 00:07:55.858070 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.858041 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:07:55.858347 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.858309 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-kube-api-access-xhlfm" (OuterVolumeSpecName: "kube-api-access-xhlfm") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "kube-api-access-xhlfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:07:55.858347 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.858317 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:07:55.858700 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.858674 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" (UID: "6b7e702f-5e99-4ea6-844f-4b8c19eaebeb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:07:55.881067 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.881031 2586 scope.go:117] "RemoveContainer" containerID="937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821" Jul 25 00:07:55.889190 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.889170 2586 scope.go:117] "RemoveContainer" containerID="061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe" Jul 25 00:07:55.889451 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:07:55.889430 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe\": container with ID starting with 061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe not found: ID does not exist" containerID="061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe" Jul 25 00:07:55.889510 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.889461 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe"} err="failed to get container status \"061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe\": rpc error: code = NotFound desc = could not find container \"061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe\": container with ID starting with 061f9cfaa5e85870ba6e03e5361d72f91fbe5ab6400fb7b8237712f36ff1f3fe not found: ID does not exist" Jul 25 00:07:55.889510 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.889477 2586 scope.go:117] "RemoveContainer" containerID="937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821" Jul 25 00:07:55.889725 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:07:55.889706 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821\": container with ID starting with 937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821 not found: ID does not exist" containerID="937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821" Jul 25 00:07:55.889768 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.889732 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821"} err="failed to get container status \"937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821\": rpc error: code = NotFound desc = could not find container \"937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821\": container with ID starting with 937559a1196b5834ec2e02d4c22b654ac08c2a2de2f102610fed163042c6a821 not found: ID does not exist" Jul 25 00:07:55.956499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956471 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.956499 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956495 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.956658 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956505 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.956658 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956516 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.956658 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956523 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.956658 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956531 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhlfm\" (UniqueName: \"kubernetes.io/projected/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-kube-api-access-xhlfm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:55.956658 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:55.956541 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:07:56.178779 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:56.178744 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k"] Jul 25 00:07:56.182391 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:56.182356 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-97aaee-openshift-default-6b996f64d-jp27k"] Jul 25 00:07:56.787065 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:07:56.787034 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" path="/var/lib/kubelet/pods/6b7e702f-5e99-4ea6-844f-4b8c19eaebeb/volumes" Jul 25 00:08:06.433850 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.433813 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2"] Jul 25 00:08:06.434254 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.434061 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" podUID="00354262-90e1-4e1d-a7bf-e0b3e807dff9" containerName="payload-processing" containerID="cri-o://7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917" gracePeriod=30 Jul 25 00:08:06.449097 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.449074 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns"] Jul 25 00:08:06.449578 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.449308 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" podUID="e1e1dc38-569d-47ee-aafa-c9ab4a993682" containerName="payload-pre-processing" containerID="cri-o://fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd" gracePeriod=30 Jul 25 00:08:06.737593 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.737561 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:08:06.744118 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.744093 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:08:06.849819 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.849787 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e1dc38-569d-47ee-aafa-c9ab4a993682-plugins-config-volume\") pod \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " Jul 25 00:08:06.849964 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.849838 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjpdd\" (UniqueName: \"kubernetes.io/projected/00354262-90e1-4e1d-a7bf-e0b3e807dff9-kube-api-access-jjpdd\") pod \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " Jul 25 00:08:06.850009 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.849955 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lfvc\" (UniqueName: \"kubernetes.io/projected/e1e1dc38-569d-47ee-aafa-c9ab4a993682-kube-api-access-2lfvc\") pod \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\" (UID: \"e1e1dc38-569d-47ee-aafa-c9ab4a993682\") " Jul 25 00:08:06.850009 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.849991 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00354262-90e1-4e1d-a7bf-e0b3e807dff9-plugins-config-volume\") pod \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\" (UID: \"00354262-90e1-4e1d-a7bf-e0b3e807dff9\") " Jul 25 00:08:06.850255 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.850212 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e1dc38-569d-47ee-aafa-c9ab4a993682-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1e1dc38-569d-47ee-aafa-c9ab4a993682" (UID: "e1e1dc38-569d-47ee-aafa-c9ab4a993682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:08:06.850391 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.850358 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e1dc38-569d-47ee-aafa-c9ab4a993682-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:06.850391 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.850357 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00354262-90e1-4e1d-a7bf-e0b3e807dff9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00354262-90e1-4e1d-a7bf-e0b3e807dff9" (UID: "00354262-90e1-4e1d-a7bf-e0b3e807dff9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:08:06.851915 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.851887 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00354262-90e1-4e1d-a7bf-e0b3e807dff9-kube-api-access-jjpdd" (OuterVolumeSpecName: "kube-api-access-jjpdd") pod "00354262-90e1-4e1d-a7bf-e0b3e807dff9" (UID: "00354262-90e1-4e1d-a7bf-e0b3e807dff9"). InnerVolumeSpecName "kube-api-access-jjpdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:08:06.852015 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.851979 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e1dc38-569d-47ee-aafa-c9ab4a993682-kube-api-access-2lfvc" (OuterVolumeSpecName: "kube-api-access-2lfvc") pod "e1e1dc38-569d-47ee-aafa-c9ab4a993682" (UID: "e1e1dc38-569d-47ee-aafa-c9ab4a993682"). InnerVolumeSpecName "kube-api-access-2lfvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:08:06.913555 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.913520 2586 generic.go:356] "Generic (PLEG): container finished" podID="00354262-90e1-4e1d-a7bf-e0b3e807dff9" containerID="7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917" exitCode=0 Jul 25 00:08:06.913694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.913565 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" Jul 25 00:08:06.913694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.913600 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" event={"ID":"00354262-90e1-4e1d-a7bf-e0b3e807dff9","Type":"ContainerDied","Data":"7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917"} Jul 25 00:08:06.913694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.913667 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2" event={"ID":"00354262-90e1-4e1d-a7bf-e0b3e807dff9","Type":"ContainerDied","Data":"151999ab861058c546e7c57f9fc300d019f3c874d7b9b970f059efa870f98e85"} Jul 25 00:08:06.913837 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.913701 2586 scope.go:117] "RemoveContainer" containerID="7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917" Jul 25 00:08:06.914950 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.914928 2586 generic.go:356] "Generic (PLEG): container finished" podID="e1e1dc38-569d-47ee-aafa-c9ab4a993682" containerID="fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd" exitCode=0 Jul 25 00:08:06.915039 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.914967 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" Jul 25 00:08:06.915039 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.915013 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" event={"ID":"e1e1dc38-569d-47ee-aafa-c9ab4a993682","Type":"ContainerDied","Data":"fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd"} Jul 25 00:08:06.915142 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.915061 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns" event={"ID":"e1e1dc38-569d-47ee-aafa-c9ab4a993682","Type":"ContainerDied","Data":"2949c1eccef595cb3ec386e4e6f76dc89ff3bf576e2ef5abb908b15460ac4261"} Jul 25 00:08:06.922872 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.922836 2586 scope.go:117] "RemoveContainer" containerID="7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917" Jul 25 00:08:06.923117 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:08:06.923095 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917\": container with ID starting with 7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917 not found: ID does not exist" containerID="7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917" Jul 25 00:08:06.923205 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.923123 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917"} err="failed to get container status \"7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917\": rpc error: code = NotFound desc = could not find container \"7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917\": container with ID starting with 7115934983d9238d158e2a15956a2926662cb0041d08345a125656c53ac5f917 not found: ID does not exist" Jul 25 00:08:06.923205 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.923143 2586 scope.go:117] "RemoveContainer" containerID="fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd" Jul 25 00:08:06.931366 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.931351 2586 scope.go:117] "RemoveContainer" containerID="fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd" Jul 25 00:08:06.931615 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:08:06.931597 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd\": container with ID starting with fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd not found: ID does not exist" containerID="fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd" Jul 25 00:08:06.931677 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.931621 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd"} err="failed to get container status \"fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd\": rpc error: code = NotFound desc = could not find container \"fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd\": container with ID starting with fb75c5e3d721d148c4b498ddede780ca2822287eee6fb13ce63c59cf5ea11fdd not found: ID does not exist" Jul 25 00:08:06.939413 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.939345 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2"] Jul 25 00:08:06.941699 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.941676 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-7d3cec-79477d979b-75wn2"] Jul 25 00:08:06.950888 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.950868 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns"] Jul 25 00:08:06.951243 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.951218 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjpdd\" (UniqueName: \"kubernetes.io/projected/00354262-90e1-4e1d-a7bf-e0b3e807dff9-kube-api-access-jjpdd\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:06.951354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.951265 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfvc\" (UniqueName: \"kubernetes.io/projected/e1e1dc38-569d-47ee-aafa-c9ab4a993682-kube-api-access-2lfvc\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:06.951354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.951280 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00354262-90e1-4e1d-a7bf-e0b3e807dff9-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:06.954035 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:06.954013 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7d3cec-7d8cb7d49b-p7bns"] Jul 25 00:08:08.786967 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:08.786933 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00354262-90e1-4e1d-a7bf-e0b3e807dff9" path="/var/lib/kubelet/pods/00354262-90e1-4e1d-a7bf-e0b3e807dff9/volumes" Jul 25 00:08:08.787393 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:08.787374 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e1dc38-569d-47ee-aafa-c9ab4a993682" path="/var/lib/kubelet/pods/e1e1dc38-569d-47ee-aafa-c9ab4a993682/volumes" Jul 25 00:08:17.409272 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:17.405850 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94"] Jul 25 00:08:17.409272 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:17.406101 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" containerID="cri-o://cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6" gracePeriod=30 Jul 25 00:08:18.329713 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.329674 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7"] Jul 25 00:08:18.330416 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330400 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" Jul 25 00:08:18.330467 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330420 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" Jul 25 00:08:18.330467 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330446 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" Jul 25 00:08:18.330467 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330456 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" Jul 25 00:08:18.330576 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330468 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00354262-90e1-4e1d-a7bf-e0b3e807dff9" containerName="payload-processing" Jul 25 00:08:18.330576 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330477 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="00354262-90e1-4e1d-a7bf-e0b3e807dff9" containerName="payload-processing" Jul 25 00:08:18.330576 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330491 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e1dc38-569d-47ee-aafa-c9ab4a993682" containerName="payload-pre-processing" Jul 25 00:08:18.330576 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330500 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e1dc38-569d-47ee-aafa-c9ab4a993682" containerName="payload-pre-processing" Jul 25 00:08:18.330694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330598 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="00354262-90e1-4e1d-a7bf-e0b3e807dff9" containerName="payload-processing" Jul 25 00:08:18.330694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330610 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e1dc38-569d-47ee-aafa-c9ab4a993682" containerName="payload-pre-processing" Jul 25 00:08:18.330694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330620 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" Jul 25 00:08:18.330694 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.330630 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b7e702f-5e99-4ea6-844f-4b8c19eaebeb" containerName="istio-proxy" Jul 25 00:08:18.334313 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.334285 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.336999 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.336779 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-92df89-openshift-default-dockercfg-svpj4\"" Jul 25 00:08:18.344615 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.344212 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7"] Jul 25 00:08:18.450892 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.450854 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-credential-socket\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.450892 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.450894 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-podinfo\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451033 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-token\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451073 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-data\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451103 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-envoy\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451121 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701381fd-057b-4f2b-be78-d874d5d0deaa-istiod-ca-cert\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451142 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z97xs\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-kube-api-access-z97xs\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451192 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-certs\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.451485 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.451230 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-socket\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552326 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-token\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552379 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-data\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552398 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-envoy\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552415 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701381fd-057b-4f2b-be78-d874d5d0deaa-istiod-ca-cert\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552433 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z97xs\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-kube-api-access-z97xs\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552466 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-certs\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552511 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-socket\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552571 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-credential-socket\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.552651 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552598 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-podinfo\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.553059 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552905 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-data\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.553059 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.552970 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-socket\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.553162 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.553121 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-certs\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.553216 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.553196 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-credential-socket\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.553613 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.553587 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701381fd-057b-4f2b-be78-d874d5d0deaa-istiod-ca-cert\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.554900 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.554880 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-envoy\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.555048 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.555028 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-podinfo\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.560772 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.560743 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z97xs\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-kube-api-access-z97xs\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.560925 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.560904 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-token\") pod \"e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.652640 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.652606 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:18.803744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.803714 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7"] Jul 25 00:08:18.805640 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:08:18.805613 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod701381fd_057b_4f2b_be78_d874d5d0deaa.slice/crio-b0313d09029ba9dcaf239f95c0d840fe0e984e7a73a7ec5e805758973744b092 WatchSource:0}: Error finding container b0313d09029ba9dcaf239f95c0d840fe0e984e7a73a7ec5e805758973744b092: Status 404 returned error can't find the container with id b0313d09029ba9dcaf239f95c0d840fe0e984e7a73a7ec5e805758973744b092 Jul 25 00:08:18.807790 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.807758 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:18.807870 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.807827 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:18.807870 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.807864 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:18.984662 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.984573 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" event={"ID":"701381fd-057b-4f2b-be78-d874d5d0deaa","Type":"ContainerStarted","Data":"0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179"} Jul 25 00:08:18.984662 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:18.984632 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" event={"ID":"701381fd-057b-4f2b-be78-d874d5d0deaa","Type":"ContainerStarted","Data":"b0313d09029ba9dcaf239f95c0d840fe0e984e7a73a7ec5e805758973744b092"} Jul 25 00:08:19.011219 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:19.011165 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podStartSLOduration=1.011150456 podStartE2EDuration="1.011150456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:08:19.00173507 +0000 UTC m=+2358.694575898" watchObservedRunningTime="2026-07-25 00:08:19.011150456 +0000 UTC m=+2358.703991284" Jul 25 00:08:19.653079 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:19.653041 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:19.658321 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:19.658294 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:19.991141 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:19.991053 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:19.992024 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:19.992000 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:20.075266 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:20.075223 2586 patch_prober.go:28] interesting pod/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 25 00:08:20.075409 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:20.075288 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 25 00:08:22.689020 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.688950 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:08:22.795431 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795400 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-certs\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795462 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-data\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795481 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7664940-a25f-4fdf-9be9-e0ef71e85426-istiod-ca-cert\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795502 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-socket\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795518 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-podinfo\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795532 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-credential-socket\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795560 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkhlb\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-kube-api-access-wkhlb\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795606 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795602 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-token\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795642 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-envoy\") pod \"a7664940-a25f-4fdf-9be9-e0ef71e85426\" (UID: \"a7664940-a25f-4fdf-9be9-e0ef71e85426\") " Jul 25 00:08:22.795966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795740 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:08:22.795966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795855 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:08:22.795966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795867 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:08:22.795966 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.795941 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7664940-a25f-4fdf-9be9-e0ef71e85426-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:08:22.796184 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.796000 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.796184 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.796021 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.796184 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.796034 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.796894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.796862 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-data" (OuterVolumeSpecName: "istio-data") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:08:22.798058 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.798029 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:08:22.798058 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.798044 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-token" (OuterVolumeSpecName: "istio-token") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:08:22.798203 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.798066 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:08:22.798203 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.798083 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-kube-api-access-wkhlb" (OuterVolumeSpecName: "kube-api-access-wkhlb") pod "a7664940-a25f-4fdf-9be9-e0ef71e85426" (UID: "a7664940-a25f-4fdf-9be9-e0ef71e85426"). InnerVolumeSpecName "kube-api-access-wkhlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:08:22.897040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.897001 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.897040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.897030 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a7664940-a25f-4fdf-9be9-e0ef71e85426-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.897040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.897040 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.897040 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.897048 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhlb\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-kube-api-access-wkhlb\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.897354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.897058 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:22.897354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:22.897066 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a7664940-a25f-4fdf-9be9-e0ef71e85426-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:08:23.008044 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.007945 2586 generic.go:356] "Generic (PLEG): container finished" podID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerID="cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6" exitCode=0 Jul 25 00:08:23.008044 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.008018 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" event={"ID":"a7664940-a25f-4fdf-9be9-e0ef71e85426","Type":"ContainerDied","Data":"cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6"} Jul 25 00:08:23.008044 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.008029 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" Jul 25 00:08:23.008270 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.008072 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94" event={"ID":"a7664940-a25f-4fdf-9be9-e0ef71e85426","Type":"ContainerDied","Data":"022d1ef08fd8737ca5ae2ae39faf97028fa21eda8cea1644047fb0ec6e39e946"} Jul 25 00:08:23.008270 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.008101 2586 scope.go:117] "RemoveContainer" containerID="cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6" Jul 25 00:08:23.018316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.018293 2586 scope.go:117] "RemoveContainer" containerID="de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8" Jul 25 00:08:23.026835 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.026808 2586 scope.go:117] "RemoveContainer" containerID="cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6" Jul 25 00:08:23.027136 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:08:23.027111 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6\": container with ID starting with cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6 not found: ID does not exist" containerID="cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6" Jul 25 00:08:23.027258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.027151 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6"} err="failed to get container status \"cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6\": rpc error: code = NotFound desc = could not find container \"cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6\": container with ID starting with cd34968a192b3fd3f1927bbd9b0b6a77f5696b5067f2db8feba4ba8829802dd6 not found: ID does not exist" Jul 25 00:08:23.027258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.027176 2586 scope.go:117] "RemoveContainer" containerID="de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8" Jul 25 00:08:23.027616 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:08:23.027590 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8\": container with ID starting with de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8 not found: ID does not exist" containerID="de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8" Jul 25 00:08:23.027721 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.027622 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8"} err="failed to get container status \"de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8\": rpc error: code = NotFound desc = could not find container \"de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8\": container with ID starting with de87889cb21e3811829d4fe382102c9869cb3cee30326d9a9da4862cb0db7fe8 not found: ID does not exist" Jul 25 00:08:23.029785 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.029766 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94"] Jul 25 00:08:23.033507 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:23.033482 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-7d3cec-openshift-default-7b55866fd8-pmr94"] Jul 25 00:08:24.787742 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:24.787695 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" path="/var/lib/kubelet/pods/a7664940-a25f-4fdf-9be9-e0ef71e85426/volumes" Jul 25 00:08:26.119285 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.119250 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq"] Jul 25 00:08:26.119765 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.119753 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" Jul 25 00:08:26.119830 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.119769 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" Jul 25 00:08:26.119884 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.119851 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" Jul 25 00:08:26.119884 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.119865 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" Jul 25 00:08:26.123322 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.123296 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.125619 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.125597 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-413844-openshift-default-dockercfg-2ffhx\"" Jul 25 00:08:26.133458 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.133434 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq"] Jul 25 00:08:26.224693 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224655 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-socket\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.224891 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224698 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-podinfo\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.224891 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224751 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-kube-api-access-bxt4g\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.224891 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224834 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-data\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.224891 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224864 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-credential-socket\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.225115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224893 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-certs\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.225115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.224945 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istiod-ca-cert\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.225115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.225010 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-envoy\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.225115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.225040 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-token\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326374 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326336 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-data\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326374 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326381 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-credential-socket\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326402 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-certs\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326425 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istiod-ca-cert\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326451 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-envoy\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326479 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-token\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326531 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-socket\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326554 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-podinfo\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.326653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326593 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-kube-api-access-bxt4g\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.327003 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326854 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-data\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.327003 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.326894 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-credential-socket\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.327182 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.327158 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-certs\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.327340 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.327201 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-socket\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.327340 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.327312 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istiod-ca-cert\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.329001 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.328970 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-envoy\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.329159 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.329140 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-podinfo\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.334572 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.334537 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-token\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.334730 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.334710 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-kube-api-access-bxt4g\") pod \"e2e-shared-b-413844-openshift-default-67766bf844-97tzq\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.438027 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.437927 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:26.586065 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.586011 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq"] Jul 25 00:08:26.588581 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:08:26.588537 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f6709f_5b1c_44e6_85e3_b9ebbcc0a5c5.slice/crio-2bf7c2ff5b4471da25342d06f264119aa7a1e737a3c3822e52c2d6b935c5912a WatchSource:0}: Error finding container 2bf7c2ff5b4471da25342d06f264119aa7a1e737a3c3822e52c2d6b935c5912a: Status 404 returned error can't find the container with id 2bf7c2ff5b4471da25342d06f264119aa7a1e737a3c3822e52c2d6b935c5912a Jul 25 00:08:26.590736 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.590704 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:26.590836 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.590770 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:26.590836 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:26.590796 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:27.029952 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:27.029916 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" event={"ID":"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5","Type":"ContainerStarted","Data":"4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211"} Jul 25 00:08:27.030114 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:27.029970 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" event={"ID":"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5","Type":"ContainerStarted","Data":"2bf7c2ff5b4471da25342d06f264119aa7a1e737a3c3822e52c2d6b935c5912a"} Jul 25 00:08:27.050050 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:27.049981 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" podStartSLOduration=1.049963852 podStartE2EDuration="1.049963852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:08:27.047552189 +0000 UTC m=+2366.740393020" watchObservedRunningTime="2026-07-25 00:08:27.049963852 +0000 UTC m=+2366.742804681" Jul 25 00:08:27.440938 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:27.440904 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:27.463609 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:27.463569 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:28.036487 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:28.036440 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:28.038300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:28.038269 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:29.808072 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.808036 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9"] Jul 25 00:08:29.808500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.808483 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" Jul 25 00:08:29.808500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.808497 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7664940-a25f-4fdf-9be9-e0ef71e85426" containerName="istio-proxy" Jul 25 00:08:29.810841 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.810824 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:29.813211 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.813186 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-413844\"" Jul 25 00:08:29.813376 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.813333 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-413844-dockercfg-97wn9\"" Jul 25 00:08:29.820867 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.820840 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9"] Jul 25 00:08:29.852165 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.852121 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km"] Jul 25 00:08:29.867384 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.867141 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:29.868029 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.867999 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km"] Jul 25 00:08:29.975628 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.975582 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:29.975628 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.975626 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqm4t\" (UniqueName: \"kubernetes.io/projected/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-kube-api-access-pqm4t\") pod \"payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:29.975861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.975647 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24801cd3-8cee-4fc4-a00e-02f9cf628e13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-413844-74879c78c8-qm4km\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:29.975861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:29.975769 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntvm\" (UniqueName: \"kubernetes.io/projected/24801cd3-8cee-4fc4-a00e-02f9cf628e13-kube-api-access-nntvm\") pod \"payload-processing-e2e-shared-b-413844-74879c78c8-qm4km\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:30.076486 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.076378 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:30.076486 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.076428 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqm4t\" (UniqueName: \"kubernetes.io/projected/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-kube-api-access-pqm4t\") pod \"payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:30.076486 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.076459 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24801cd3-8cee-4fc4-a00e-02f9cf628e13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-413844-74879c78c8-qm4km\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:30.076802 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.076621 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntvm\" (UniqueName: \"kubernetes.io/projected/24801cd3-8cee-4fc4-a00e-02f9cf628e13-kube-api-access-nntvm\") pod \"payload-processing-e2e-shared-b-413844-74879c78c8-qm4km\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:30.077171 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.077146 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:30.077306 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.077194 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24801cd3-8cee-4fc4-a00e-02f9cf628e13-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-413844-74879c78c8-qm4km\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:30.083682 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.083658 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqm4t\" (UniqueName: \"kubernetes.io/projected/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-kube-api-access-pqm4t\") pod \"payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:30.083788 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.083662 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntvm\" (UniqueName: \"kubernetes.io/projected/24801cd3-8cee-4fc4-a00e-02f9cf628e13-kube-api-access-nntvm\") pod \"payload-processing-e2e-shared-b-413844-74879c78c8-qm4km\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:30.122600 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.122563 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:30.180837 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.180799 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:30.325935 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.325902 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9"] Jul 25 00:08:30.329132 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:08:30.329053 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ef2a5a_c1fe_4217_81fb_36304f9f0f62.slice/crio-8f0786151cedc649f7315b0d41a48325911ebe5e96bfeae8837230f0ee8a3d0b WatchSource:0}: Error finding container 8f0786151cedc649f7315b0d41a48325911ebe5e96bfeae8837230f0ee8a3d0b: Status 404 returned error can't find the container with id 8f0786151cedc649f7315b0d41a48325911ebe5e96bfeae8837230f0ee8a3d0b Jul 25 00:08:30.400122 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:30.400042 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km"] Jul 25 00:08:30.402953 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:08:30.402904 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24801cd3_8cee_4fc4_a00e_02f9cf628e13.slice/crio-1ccb173dbbac7ef9075313e21e6bbe7d916d7482f9500016a7120ab5903b533f WatchSource:0}: Error finding container 1ccb173dbbac7ef9075313e21e6bbe7d916d7482f9500016a7120ab5903b533f: Status 404 returned error can't find the container with id 1ccb173dbbac7ef9075313e21e6bbe7d916d7482f9500016a7120ab5903b533f Jul 25 00:08:31.064086 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.064030 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" event={"ID":"24801cd3-8cee-4fc4-a00e-02f9cf628e13","Type":"ContainerStarted","Data":"d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9"} Jul 25 00:08:31.064615 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.064117 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" event={"ID":"24801cd3-8cee-4fc4-a00e-02f9cf628e13","Type":"ContainerStarted","Data":"1ccb173dbbac7ef9075313e21e6bbe7d916d7482f9500016a7120ab5903b533f"} Jul 25 00:08:31.064697 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.064672 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:31.066685 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.066652 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" event={"ID":"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62","Type":"ContainerStarted","Data":"ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710"} Jul 25 00:08:31.066861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.066709 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" event={"ID":"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62","Type":"ContainerStarted","Data":"8f0786151cedc649f7315b0d41a48325911ebe5e96bfeae8837230f0ee8a3d0b"} Jul 25 00:08:31.067286 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.067264 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:31.081449 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.081397 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" podStartSLOduration=2.081378928 podStartE2EDuration="2.081378928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:08:31.078659742 +0000 UTC m=+2370.771500596" watchObservedRunningTime="2026-07-25 00:08:31.081378928 +0000 UTC m=+2370.774219760" Jul 25 00:08:31.094665 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:31.094607 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" podStartSLOduration=2.094594908 podStartE2EDuration="2.094594908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:08:31.091819265 +0000 UTC m=+2370.784660103" watchObservedRunningTime="2026-07-25 00:08:31.094594908 +0000 UTC m=+2370.787435736" Jul 25 00:08:34.992571 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:34.992527 2586 patch_prober.go:28] interesting pod/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 25 00:08:34.993005 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:34.992592 2586 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 25 00:08:35.091919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:35.091883 2586 generic.go:356] "Generic (PLEG): container finished" podID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerID="0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179" exitCode=0 Jul 25 00:08:35.092127 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:35.091960 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" event={"ID":"701381fd-057b-4f2b-be78-d874d5d0deaa","Type":"ContainerDied","Data":"0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179"} Jul 25 00:08:35.092491 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:35.092473 2586 scope.go:117] "RemoveContainer" containerID="0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179" Jul 25 00:08:35.093570 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:35.093544 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:35.093629 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:35.093617 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:35.093669 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:35.093648 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:36.100934 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:36.100896 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" event={"ID":"701381fd-057b-4f2b-be78-d874d5d0deaa","Type":"ContainerStarted","Data":"5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33"} Jul 25 00:08:36.653388 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:36.653351 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:36.653779 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:36.653694 2586 patch_prober.go:28] interesting pod/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 25 00:08:36.653925 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:36.653902 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 25 00:08:37.653427 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:37.653384 2586 patch_prober.go:28] interesting pod/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 25 00:08:37.654071 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:37.654036 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 25 00:08:38.653281 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:38.653218 2586 patch_prober.go:28] interesting pod/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 25 00:08:38.653568 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:38.653535 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 25 00:08:39.653492 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:39.653441 2586 patch_prober.go:28] interesting pod/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 25 00:08:39.653698 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:39.653514 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 25 00:08:40.656850 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:40.656819 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:40.657349 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:40.657209 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:40.657925 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:40.657906 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:08:42.075473 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.075377 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:08:42.148203 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.148164 2586 generic.go:356] "Generic (PLEG): container finished" podID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerID="4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211" exitCode=0 Jul 25 00:08:42.148394 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.148262 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" event={"ID":"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5","Type":"ContainerDied","Data":"4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211"} Jul 25 00:08:42.148758 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.148741 2586 scope.go:117] "RemoveContainer" containerID="4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211" Jul 25 00:08:42.149742 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.149713 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:42.149806 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.149789 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:42.149854 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:42.149841 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:08:43.036971 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:43.036926 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:43.081750 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:43.081722 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:08:43.165545 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:43.165508 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" event={"ID":"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5","Type":"ContainerStarted","Data":"15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2"} Jul 25 00:08:43.439115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:43.439077 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:43.440630 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:43.440597 2586 patch_prober.go:28] interesting pod/e2e-shared-b-413844-openshift-default-67766bf844-97tzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 25 00:08:43.440782 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:43.440667 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 25 00:08:44.438433 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:44.438393 2586 patch_prober.go:28] interesting pod/e2e-shared-b-413844-openshift-default-67766bf844-97tzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 25 00:08:44.438934 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:44.438457 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 25 00:08:45.438533 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:45.438484 2586 patch_prober.go:28] interesting pod/e2e-shared-b-413844-openshift-default-67766bf844-97tzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 25 00:08:45.439065 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:45.438556 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 25 00:08:46.438932 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:46.438889 2586 patch_prober.go:28] interesting pod/e2e-shared-b-413844-openshift-default-67766bf844-97tzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 25 00:08:46.439477 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:46.438975 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 25 00:08:47.444383 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:47.444349 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:47.444846 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:47.444627 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:08:47.445525 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:08:47.445505 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:09:00.958396 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:09:00.958355 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:09:00.963144 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:09:00.963117 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:13:30.275969 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.275924 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p"] Jul 25 00:13:30.279507 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.279486 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.282733 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.282700 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-92df89\"/\"kube-root-ca.crt\"" Jul 25 00:13:30.282861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.282761 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-92df89\"/\"default-dockercfg-mqhrp\"" Jul 25 00:13:30.282861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.282772 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-92df89\"/\"sub-test-model-92df89-kserve-self-signed-certs\"" Jul 25 00:13:30.282861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.282801 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-92df89\"/\"openshift-service-ca.crt\"" Jul 25 00:13:30.288563 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.288535 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p"] Jul 25 00:13:30.393728 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.393693 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005e7420-7e99-4123-9d90-00584525d2e9-tls-certs\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.393907 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.393737 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-tmp-dir\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.393907 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.393805 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-home\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.393982 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.393927 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtbbr\" (UniqueName: \"kubernetes.io/projected/005e7420-7e99-4123-9d90-00584525d2e9-kube-api-access-mtbbr\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.394017 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.393983 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-model-cache\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.394052 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.394021 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-dshm\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.495517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.495471 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-dshm\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.495517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.495520 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005e7420-7e99-4123-9d90-00584525d2e9-tls-certs\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.495739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.495540 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-tmp-dir\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.495739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.495582 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-home\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.495739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.495629 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtbbr\" (UniqueName: \"kubernetes.io/projected/005e7420-7e99-4123-9d90-00584525d2e9-kube-api-access-mtbbr\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.495739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.495690 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-model-cache\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.496049 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.496028 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-tmp-dir\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.496108 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.496062 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-model-cache\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.496158 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.496117 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-home\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.497889 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.497867 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-dshm\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.498296 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.498275 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005e7420-7e99-4123-9d90-00584525d2e9-tls-certs\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.503805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.503778 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtbbr\" (UniqueName: \"kubernetes.io/projected/005e7420-7e99-4123-9d90-00584525d2e9-kube-api-access-mtbbr\") pod \"sub-test-model-92df89-kserve-5db77b99d7-sp64p\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.589713 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.589614 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:30.735554 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:30.735523 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p"] Jul 25 00:13:30.737791 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:13:30.737741 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod005e7420_7e99_4123_9d90_00584525d2e9.slice/crio-652259f3852a5f91760dd3d77b0663fcdf1dc0114302bd5a66779ea1ffda154c WatchSource:0}: Error finding container 652259f3852a5f91760dd3d77b0663fcdf1dc0114302bd5a66779ea1ffda154c: Status 404 returned error can't find the container with id 652259f3852a5f91760dd3d77b0663fcdf1dc0114302bd5a66779ea1ffda154c Jul 25 00:13:31.685046 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:31.685006 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" event={"ID":"005e7420-7e99-4123-9d90-00584525d2e9","Type":"ContainerStarted","Data":"ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46"} Jul 25 00:13:31.685046 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:31.685050 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" event={"ID":"005e7420-7e99-4123-9d90-00584525d2e9","Type":"ContainerStarted","Data":"652259f3852a5f91760dd3d77b0663fcdf1dc0114302bd5a66779ea1ffda154c"} Jul 25 00:13:31.704377 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:31.704321 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" podStartSLOduration=1.7043061210000001 podStartE2EDuration="1.704306121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:13:31.702152677 +0000 UTC m=+2671.394993504" watchObservedRunningTime="2026-07-25 00:13:31.704306121 +0000 UTC m=+2671.397147007" Jul 25 00:13:40.590411 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:40.590374 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:40.590411 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:40.590419 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:40.602996 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:40.602968 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:13:40.745333 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:13:40.745293 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:14:00.997415 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:14:00.997266 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:14:01.005634 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:14:01.005608 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:15:09.275921 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:09.275888 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p"] Jul 25 00:15:24.300997 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.300794 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" podUID="005e7420-7e99-4123-9d90-00584525d2e9" containerName="main" containerID="cri-o://ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46" gracePeriod=45 Jul 25 00:15:24.569405 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.569379 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:15:24.620218 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620184 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-home\") pod \"005e7420-7e99-4123-9d90-00584525d2e9\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " Jul 25 00:15:24.620218 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620225 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-tmp-dir\") pod \"005e7420-7e99-4123-9d90-00584525d2e9\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " Jul 25 00:15:24.620546 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620266 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-dshm\") pod \"005e7420-7e99-4123-9d90-00584525d2e9\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " Jul 25 00:15:24.620546 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620330 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005e7420-7e99-4123-9d90-00584525d2e9-tls-certs\") pod \"005e7420-7e99-4123-9d90-00584525d2e9\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " Jul 25 00:15:24.620546 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620350 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mtbbr\" (UniqueName: \"kubernetes.io/projected/005e7420-7e99-4123-9d90-00584525d2e9-kube-api-access-mtbbr\") pod \"005e7420-7e99-4123-9d90-00584525d2e9\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " Jul 25 00:15:24.620546 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620374 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-model-cache\") pod \"005e7420-7e99-4123-9d90-00584525d2e9\" (UID: \"005e7420-7e99-4123-9d90-00584525d2e9\") " Jul 25 00:15:24.620546 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620523 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-home" (OuterVolumeSpecName: "home") pod "005e7420-7e99-4123-9d90-00584525d2e9" (UID: "005e7420-7e99-4123-9d90-00584525d2e9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:15:24.620805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620582 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "005e7420-7e99-4123-9d90-00584525d2e9" (UID: "005e7420-7e99-4123-9d90-00584525d2e9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:15:24.620805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620693 2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-home\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:15:24.620805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620731 2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-tmp-dir\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:15:24.620805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.620726 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-model-cache" (OuterVolumeSpecName: "model-cache") pod "005e7420-7e99-4123-9d90-00584525d2e9" (UID: "005e7420-7e99-4123-9d90-00584525d2e9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:15:24.622588 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.622560 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-dshm" (OuterVolumeSpecName: "dshm") pod "005e7420-7e99-4123-9d90-00584525d2e9" (UID: "005e7420-7e99-4123-9d90-00584525d2e9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:15:24.622670 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.622626 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/005e7420-7e99-4123-9d90-00584525d2e9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "005e7420-7e99-4123-9d90-00584525d2e9" (UID: "005e7420-7e99-4123-9d90-00584525d2e9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 00:15:24.622709 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.622654 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005e7420-7e99-4123-9d90-00584525d2e9-kube-api-access-mtbbr" (OuterVolumeSpecName: "kube-api-access-mtbbr") pod "005e7420-7e99-4123-9d90-00584525d2e9" (UID: "005e7420-7e99-4123-9d90-00584525d2e9"). InnerVolumeSpecName "kube-api-access-mtbbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:15:24.722334 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.722292 2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-dshm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:15:24.722334 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.722329 2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/005e7420-7e99-4123-9d90-00584525d2e9-tls-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:15:24.722556 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.722344 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtbbr\" (UniqueName: \"kubernetes.io/projected/005e7420-7e99-4123-9d90-00584525d2e9-kube-api-access-mtbbr\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:15:24.722556 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:24.722357 2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/005e7420-7e99-4123-9d90-00584525d2e9-model-cache\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:15:25.283367 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.283330 2586 generic.go:356] "Generic (PLEG): container finished" podID="005e7420-7e99-4123-9d90-00584525d2e9" containerID="ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46" exitCode=0 Jul 25 00:15:25.283624 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.283374 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" Jul 25 00:15:25.283624 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.283420 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" event={"ID":"005e7420-7e99-4123-9d90-00584525d2e9","Type":"ContainerDied","Data":"ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46"} Jul 25 00:15:25.283624 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.283468 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p" event={"ID":"005e7420-7e99-4123-9d90-00584525d2e9","Type":"ContainerDied","Data":"652259f3852a5f91760dd3d77b0663fcdf1dc0114302bd5a66779ea1ffda154c"} Jul 25 00:15:25.283624 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.283490 2586 scope.go:117] "RemoveContainer" containerID="ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46" Jul 25 00:15:25.292908 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.292888 2586 scope.go:117] "RemoveContainer" containerID="ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46" Jul 25 00:15:25.293219 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:15:25.293198 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46\": container with ID starting with ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46 not found: ID does not exist" containerID="ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46" Jul 25 00:15:25.293312 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.293228 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46"} err="failed to get container status \"ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46\": rpc error: code = NotFound desc = could not find container \"ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46\": container with ID starting with ea2d59084054003dc50967fb0f8c12b5be67153e47114fba0f18201d917d2c46 not found: ID does not exist" Jul 25 00:15:25.299648 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.299623 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p"] Jul 25 00:15:25.305043 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:25.305016 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-92df89/sub-test-model-92df89-kserve-5db77b99d7-sp64p"] Jul 25 00:15:26.787449 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:15:26.787418 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="005e7420-7e99-4123-9d90-00584525d2e9" path="/var/lib/kubelet/pods/005e7420-7e99-4123-9d90-00584525d2e9/volumes" Jul 25 00:16:07.897851 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.897812 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4"] Jul 25 00:16:07.898359 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.898274 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="005e7420-7e99-4123-9d90-00584525d2e9" containerName="main" Jul 25 00:16:07.898359 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.898287 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="005e7420-7e99-4123-9d90-00584525d2e9" containerName="main" Jul 25 00:16:07.898440 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.898369 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="005e7420-7e99-4123-9d90-00584525d2e9" containerName="main" Jul 25 00:16:07.902836 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.902816 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.906894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.906515 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-413844\"/\"rate-test-model-413844-kserve-self-signed-certs\"" Jul 25 00:16:07.906894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.906539 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-413844\"/\"default-dockercfg-gzckw\"" Jul 25 00:16:07.906894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.906566 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-413844\"/\"openshift-service-ca.crt\"" Jul 25 00:16:07.907437 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.907418 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-413844\"/\"kube-root-ca.crt\"" Jul 25 00:16:07.909519 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.909496 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-home\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.909632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.909527 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-model-cache\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.909632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.909556 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9md4\" (UniqueName: \"kubernetes.io/projected/abb4c308-3f25-4210-b9a5-0459c20399a7-kube-api-access-x9md4\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.909632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.909618 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-tmp-dir\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.909782 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.909684 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-dshm\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.909782 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.909715 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb4c308-3f25-4210-b9a5-0459c20399a7-tls-certs\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:07.910159 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:07.910136 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4"] Jul 25 00:16:08.010555 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010522 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-dshm\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.010555 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010565 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb4c308-3f25-4210-b9a5-0459c20399a7-tls-certs\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.010786 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010598 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-home\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.010786 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010622 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-model-cache\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.010786 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010660 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9md4\" (UniqueName: \"kubernetes.io/projected/abb4c308-3f25-4210-b9a5-0459c20399a7-kube-api-access-x9md4\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.010786 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010721 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-tmp-dir\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.011013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.010987 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-model-cache\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.011070 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.011043 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-home\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.011122 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.011103 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-tmp-dir\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.012904 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.012881 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-dshm\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.013191 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.013168 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb4c308-3f25-4210-b9a5-0459c20399a7-tls-certs\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.019343 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.019321 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9md4\" (UniqueName: \"kubernetes.io/projected/abb4c308-3f25-4210-b9a5-0459c20399a7-kube-api-access-x9md4\") pod \"rate-test-model-413844-kserve-55869c5755-mpkw4\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.213847 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.213767 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:08.347825 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.347800 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4"] Jul 25 00:16:08.349890 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:16:08.349861 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabb4c308_3f25_4210_b9a5_0459c20399a7.slice/crio-a66561bc2c7a2cd7e1f8ac7b8818351524704104cb6147c92193fec29342f8f7 WatchSource:0}: Error finding container a66561bc2c7a2cd7e1f8ac7b8818351524704104cb6147c92193fec29342f8f7: Status 404 returned error can't find the container with id a66561bc2c7a2cd7e1f8ac7b8818351524704104cb6147c92193fec29342f8f7 Jul 25 00:16:08.508504 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.508419 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" event={"ID":"abb4c308-3f25-4210-b9a5-0459c20399a7","Type":"ContainerStarted","Data":"d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e"} Jul 25 00:16:08.508504 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.508466 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" event={"ID":"abb4c308-3f25-4210-b9a5-0459c20399a7","Type":"ContainerStarted","Data":"a66561bc2c7a2cd7e1f8ac7b8818351524704104cb6147c92193fec29342f8f7"} Jul 25 00:16:08.523442 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:08.523397 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" podStartSLOduration=1.523379866 podStartE2EDuration="1.523379866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:16:08.5231842 +0000 UTC m=+2828.216025033" watchObservedRunningTime="2026-07-25 00:16:08.523379866 +0000 UTC m=+2828.216220701" Jul 25 00:16:18.214915 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:18.214873 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:18.215418 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:18.214954 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:18.227693 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:18.227672 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:16:18.573209 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:16:18.573126 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:17:23.594675 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:23.594637 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4"] Jul 25 00:17:38.615045 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.614992 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" podUID="abb4c308-3f25-4210-b9a5-0459c20399a7" containerName="main" containerID="cri-o://d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e" gracePeriod=45 Jul 25 00:17:38.876124 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.876063 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:17:38.968337 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968305 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb4c308-3f25-4210-b9a5-0459c20399a7-tls-certs\") pod \"abb4c308-3f25-4210-b9a5-0459c20399a7\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " Jul 25 00:17:38.968337 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968341 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-home\") pod \"abb4c308-3f25-4210-b9a5-0459c20399a7\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " Jul 25 00:17:38.968542 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968395 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9md4\" (UniqueName: \"kubernetes.io/projected/abb4c308-3f25-4210-b9a5-0459c20399a7-kube-api-access-x9md4\") pod \"abb4c308-3f25-4210-b9a5-0459c20399a7\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " Jul 25 00:17:38.968542 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968429 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-model-cache\") pod \"abb4c308-3f25-4210-b9a5-0459c20399a7\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " Jul 25 00:17:38.968542 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968451 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-dshm\") pod \"abb4c308-3f25-4210-b9a5-0459c20399a7\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " Jul 25 00:17:38.968542 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968500 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-tmp-dir\") pod \"abb4c308-3f25-4210-b9a5-0459c20399a7\" (UID: \"abb4c308-3f25-4210-b9a5-0459c20399a7\") " Jul 25 00:17:38.968752 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968692 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-home" (OuterVolumeSpecName: "home") pod "abb4c308-3f25-4210-b9a5-0459c20399a7" (UID: "abb4c308-3f25-4210-b9a5-0459c20399a7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:17:38.968752 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968726 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-model-cache" (OuterVolumeSpecName: "model-cache") pod "abb4c308-3f25-4210-b9a5-0459c20399a7" (UID: "abb4c308-3f25-4210-b9a5-0459c20399a7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:17:38.968844 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968799 2586 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-home\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:17:38.968881 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.968848 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "abb4c308-3f25-4210-b9a5-0459c20399a7" (UID: "abb4c308-3f25-4210-b9a5-0459c20399a7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:17:38.970584 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.970563 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abb4c308-3f25-4210-b9a5-0459c20399a7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "abb4c308-3f25-4210-b9a5-0459c20399a7" (UID: "abb4c308-3f25-4210-b9a5-0459c20399a7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 25 00:17:38.970654 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.970579 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abb4c308-3f25-4210-b9a5-0459c20399a7-kube-api-access-x9md4" (OuterVolumeSpecName: "kube-api-access-x9md4") pod "abb4c308-3f25-4210-b9a5-0459c20399a7" (UID: "abb4c308-3f25-4210-b9a5-0459c20399a7"). InnerVolumeSpecName "kube-api-access-x9md4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:17:38.970654 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.970635 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-dshm" (OuterVolumeSpecName: "dshm") pod "abb4c308-3f25-4210-b9a5-0459c20399a7" (UID: "abb4c308-3f25-4210-b9a5-0459c20399a7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:17:38.993388 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.993359 2586 generic.go:356] "Generic (PLEG): container finished" podID="abb4c308-3f25-4210-b9a5-0459c20399a7" containerID="d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e" exitCode=0 Jul 25 00:17:38.993500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.993404 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" Jul 25 00:17:38.993500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.993443 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" event={"ID":"abb4c308-3f25-4210-b9a5-0459c20399a7","Type":"ContainerDied","Data":"d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e"} Jul 25 00:17:38.993500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.993483 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4" event={"ID":"abb4c308-3f25-4210-b9a5-0459c20399a7","Type":"ContainerDied","Data":"a66561bc2c7a2cd7e1f8ac7b8818351524704104cb6147c92193fec29342f8f7"} Jul 25 00:17:38.993500 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:38.993501 2586 scope.go:117] "RemoveContainer" containerID="d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e" Jul 25 00:17:39.002340 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.002324 2586 scope.go:117] "RemoveContainer" containerID="d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e" Jul 25 00:17:39.002588 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:17:39.002568 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e\": container with ID starting with d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e not found: ID does not exist" containerID="d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e" Jul 25 00:17:39.002627 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.002596 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e"} err="failed to get container status \"d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e\": rpc error: code = NotFound desc = could not find container \"d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e\": container with ID starting with d94c1db80afbdb1e5f10df44d58634220d609bff41c184757ee1e3a5ca1d340e not found: ID does not exist" Jul 25 00:17:39.013566 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.013544 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4"] Jul 25 00:17:39.018826 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.018805 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-413844/rate-test-model-413844-kserve-55869c5755-mpkw4"] Jul 25 00:17:39.069744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.069723 2586 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abb4c308-3f25-4210-b9a5-0459c20399a7-tls-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:17:39.069744 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.069744 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9md4\" (UniqueName: \"kubernetes.io/projected/abb4c308-3f25-4210-b9a5-0459c20399a7-kube-api-access-x9md4\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:17:39.069856 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.069754 2586 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-model-cache\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:17:39.069856 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.069763 2586 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-dshm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:17:39.069856 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:39.069770 2586 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/abb4c308-3f25-4210-b9a5-0459c20399a7-tmp-dir\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:17:40.788633 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:40.788594 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abb4c308-3f25-4210-b9a5-0459c20399a7" path="/var/lib/kubelet/pods/abb4c308-3f25-4210-b9a5-0459c20399a7/volumes" Jul 25 00:17:43.982609 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:43.982577 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg"] Jul 25 00:17:43.983064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:43.983016 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abb4c308-3f25-4210-b9a5-0459c20399a7" containerName="main" Jul 25 00:17:43.983064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:43.983030 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="abb4c308-3f25-4210-b9a5-0459c20399a7" containerName="main" Jul 25 00:17:43.983182 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:43.983117 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="abb4c308-3f25-4210-b9a5-0459c20399a7" containerName="main" Jul 25 00:17:43.986390 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:43.986372 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:43.988737 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:43.988716 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-be7457-openshift-default-dockercfg-ggjnh\"" Jul 25 00:17:44.005380 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.005347 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg"] Jul 25 00:17:44.108896 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.108852 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-podinfo\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.108896 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.108903 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-data\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109123 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.108925 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-token\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109123 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.108991 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pwng\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-kube-api-access-6pwng\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109123 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.109070 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-envoy\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109123 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.109105 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istiod-ca-cert\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109289 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.109123 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-credential-socket\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109289 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.109213 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-socket\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.109358 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.109319 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-certs\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.209949 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.209881 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-podinfo\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.209949 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.209957 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-data\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.209992 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-token\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210025 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pwng\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-kube-api-access-6pwng\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210063 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-envoy\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210093 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istiod-ca-cert\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210119 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-credential-socket\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210176 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-socket\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210259 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-certs\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210737 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210525 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-data\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210737 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210703 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-certs\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210850 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210794 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-credential-socket\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210914 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210887 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-socket\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.210970 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.210919 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istiod-ca-cert\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.212555 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.212530 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-envoy\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.212709 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.212692 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-podinfo\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.217673 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.217634 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-token\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.217959 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.217929 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pwng\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-kube-api-access-6pwng\") pod \"e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.308013 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.307906 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:44.453507 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.453483 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg"] Jul 25 00:17:44.460335 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.460295 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:17:44.460481 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.460412 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:17:44.460481 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:44.460458 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:17:45.026863 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:45.026817 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" event={"ID":"e254f8c6-b61e-4ff5-9dce-40fcdf82613a","Type":"ContainerStarted","Data":"cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269"} Jul 25 00:17:45.027282 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:45.026884 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" event={"ID":"e254f8c6-b61e-4ff5-9dce-40fcdf82613a","Type":"ContainerStarted","Data":"96efe0adee08ed00562c1f244da344e384e83cc4444e3fdfc6d52aec15c665d9"} Jul 25 00:17:45.048858 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:45.048802 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" podStartSLOduration=2.048786009 podStartE2EDuration="2.048786009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:17:45.046452379 +0000 UTC m=+2924.739293221" watchObservedRunningTime="2026-07-25 00:17:45.048786009 +0000 UTC m=+2924.741626909" Jul 25 00:17:45.308390 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:45.308280 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:45.313680 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:45.313649 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:46.032562 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:46.032528 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:46.033668 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:46.033644 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:17:48.737305 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.737255 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq"] Jul 25 00:17:48.741274 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.741251 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:48.743751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.743596 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-be7457\"" Jul 25 00:17:48.743751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.743613 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-be7457-dockercfg-vs7wn\"" Jul 25 00:17:48.748885 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.748862 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq"] Jul 25 00:17:48.856727 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.856686 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b4abf6-cf04-42a0-a399-6a0324e240b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:48.856910 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.856810 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qp5\" (UniqueName: \"kubernetes.io/projected/55b4abf6-cf04-42a0-a399-6a0324e240b3-kube-api-access-x8qp5\") pod \"payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:48.957629 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.957584 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b4abf6-cf04-42a0-a399-6a0324e240b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:48.957818 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.957677 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qp5\" (UniqueName: \"kubernetes.io/projected/55b4abf6-cf04-42a0-a399-6a0324e240b3-kube-api-access-x8qp5\") pod \"payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:48.958928 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.958880 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b4abf6-cf04-42a0-a399-6a0324e240b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:48.967550 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:48.967523 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qp5\" (UniqueName: \"kubernetes.io/projected/55b4abf6-cf04-42a0-a399-6a0324e240b3-kube-api-access-x8qp5\") pod \"payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:49.054582 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:49.054496 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:49.201312 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:49.201283 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq"] Jul 25 00:17:49.202789 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:17:49.202762 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b4abf6_cf04_42a0_a399_6a0324e240b3.slice/crio-24e470a15a5cfddb87cc92993a65bf035d1335703b9937371c68b18030d3ae32 WatchSource:0}: Error finding container 24e470a15a5cfddb87cc92993a65bf035d1335703b9937371c68b18030d3ae32: Status 404 returned error can't find the container with id 24e470a15a5cfddb87cc92993a65bf035d1335703b9937371c68b18030d3ae32 Jul 25 00:17:50.059674 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:50.059620 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" event={"ID":"55b4abf6-cf04-42a0-a399-6a0324e240b3","Type":"ContainerStarted","Data":"07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd"} Jul 25 00:17:50.060145 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:50.059696 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:17:50.060145 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:50.059723 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" event={"ID":"55b4abf6-cf04-42a0-a399-6a0324e240b3","Type":"ContainerStarted","Data":"24e470a15a5cfddb87cc92993a65bf035d1335703b9937371c68b18030d3ae32"} Jul 25 00:17:50.075965 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:17:50.075909 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" podStartSLOduration=2.075891758 podStartE2EDuration="2.075891758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:17:50.073268836 +0000 UTC m=+2929.766109665" watchObservedRunningTime="2026-07-25 00:17:50.075891758 +0000 UTC m=+2929.768732635" Jul 25 00:18:01.069354 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:01.069318 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:18:04.141643 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:04.141609 2586 generic.go:356] "Generic (PLEG): container finished" podID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerID="cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269" exitCode=0 Jul 25 00:18:04.142031 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:04.141679 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" event={"ID":"e254f8c6-b61e-4ff5-9dce-40fcdf82613a","Type":"ContainerDied","Data":"cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269"} Jul 25 00:18:04.142126 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:04.142112 2586 scope.go:117] "RemoveContainer" containerID="cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269" Jul 25 00:18:04.143376 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:04.143342 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:18:04.143467 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:04.143450 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:18:04.143516 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:04.143502 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:18:05.157427 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:05.157378 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" event={"ID":"e254f8c6-b61e-4ff5-9dce-40fcdf82613a","Type":"ContainerStarted","Data":"69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7"} Jul 25 00:18:05.308883 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:05.308849 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:18:05.309084 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:05.309064 2586 patch_prober.go:28] interesting pod/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 25 00:18:05.309130 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:05.309107 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 25 00:18:06.309338 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:06.309295 2586 patch_prober.go:28] interesting pod/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 25 00:18:06.309892 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:06.309361 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 25 00:18:07.309262 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:07.309199 2586 patch_prober.go:28] interesting pod/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 25 00:18:07.309447 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:07.309298 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 25 00:18:08.309316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.309279 2586 patch_prober.go:28] interesting pod/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 25 00:18:08.309517 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.309339 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 25 00:18:08.495211 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.495175 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh"] Jul 25 00:18:08.497754 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.497735 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.500640 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.500614 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-899559\"" Jul 25 00:18:08.500762 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.500615 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-899559-dockercfg-bkk58\"" Jul 25 00:18:08.508870 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.508841 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh"] Jul 25 00:18:08.666856 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.666818 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz7nc\" (UniqueName: \"kubernetes.io/projected/4ab8d029-0673-4620-9c0b-a72b92d186c1-kube-api-access-fz7nc\") pod \"payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.667050 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.666897 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8d029-0673-4620-9c0b-a72b92d186c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.767568 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.767526 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8d029-0673-4620-9c0b-a72b92d186c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.767758 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.767629 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz7nc\" (UniqueName: \"kubernetes.io/projected/4ab8d029-0673-4620-9c0b-a72b92d186c1-kube-api-access-fz7nc\") pod \"payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.768251 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.768215 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8d029-0673-4620-9c0b-a72b92d186c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.775080 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.775052 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz7nc\" (UniqueName: \"kubernetes.io/projected/4ab8d029-0673-4620-9c0b-a72b92d186c1-kube-api-access-fz7nc\") pod \"payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.810297 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.810275 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:08.996196 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:08.996164 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh"] Jul 25 00:18:08.998405 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:18:08.998378 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ab8d029_0673_4620_9c0b_a72b92d186c1.slice/crio-12dacc4751527d7da8a854c7171ab5fb916bab5089b630d2c08591309c180b9b WatchSource:0}: Error finding container 12dacc4751527d7da8a854c7171ab5fb916bab5089b630d2c08591309c180b9b: Status 404 returned error can't find the container with id 12dacc4751527d7da8a854c7171ab5fb916bab5089b630d2c08591309c180b9b Jul 25 00:18:09.183726 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.183692 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" event={"ID":"4ab8d029-0673-4620-9c0b-a72b92d186c1","Type":"ContainerStarted","Data":"0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b"} Jul 25 00:18:09.183901 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.183744 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" event={"ID":"4ab8d029-0673-4620-9c0b-a72b92d186c1","Type":"ContainerStarted","Data":"12dacc4751527d7da8a854c7171ab5fb916bab5089b630d2c08591309c180b9b"} Jul 25 00:18:09.183901 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.183784 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:18:09.198030 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.197973 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" podStartSLOduration=1.1979575279999999 podStartE2EDuration="1.197957528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:18:09.197535203 +0000 UTC m=+2948.890376032" watchObservedRunningTime="2026-07-25 00:18:09.197957528 +0000 UTC m=+2948.890798393" Jul 25 00:18:09.328958 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.328860 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:18:09.329466 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.329275 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:18:09.330056 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:09.330032 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:18:20.194156 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:18:20.194123 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:19:01.041973 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:01.041940 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:19:01.050285 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:01.050264 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:19:14.927101 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:14.927041 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq_55b4abf6-cf04-42a0-a399-6a0324e240b3/payload-processing/0.log" Jul 25 00:19:15.733830 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.733795 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl"] Jul 25 00:19:15.737575 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.737551 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.740141 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.740113 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-85930d-openshift-default-dockercfg-gq2bg\"" Jul 25 00:19:15.760384 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.760315 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl"] Jul 25 00:19:15.790392 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790364 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-certs\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790518 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790396 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-credential-socket\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790518 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790419 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9cv\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-kube-api-access-kj9cv\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790518 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790436 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-token\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790683 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790515 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-socket\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790683 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790558 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-envoy\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790683 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790609 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790683 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790674 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-podinfo\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.790824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.790704 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-data\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891451 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891407 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-podinfo\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891467 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-data\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891508 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-certs\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891532 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-credential-socket\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891567 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9cv\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-kube-api-access-kj9cv\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891600 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-token\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891664 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891651 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-socket\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891680 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-envoy\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891723 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891901 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-certs\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.891991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.891959 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-credential-socket\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.892258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.892010 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-data\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.892258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.892111 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-socket\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.892622 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.892599 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istiod-ca-cert\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.894300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.894279 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-envoy\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.894421 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.894359 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-podinfo\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.899759 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.899740 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-token\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:15.899956 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:15.899941 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9cv\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-kube-api-access-kj9cv\") pod \"e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:16.062955 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.062867 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:16.204600 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.204574 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl"] Jul 25 00:19:16.206443 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:19:16.206415 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2de5a71_1d70_4ce5_828d_babfbc31e7a5.slice/crio-66342065ac736b3f8a19458a919890999781df8a6331dbcae5993179476f2233 WatchSource:0}: Error finding container 66342065ac736b3f8a19458a919890999781df8a6331dbcae5993179476f2233: Status 404 returned error can't find the container with id 66342065ac736b3f8a19458a919890999781df8a6331dbcae5993179476f2233 Jul 25 00:19:16.208601 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.208569 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:19:16.208689 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.208638 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:19:16.208689 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.208667 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:19:16.547275 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.547225 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" event={"ID":"d2de5a71-1d70-4ce5-828d-babfbc31e7a5","Type":"ContainerStarted","Data":"9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207"} Jul 25 00:19:16.547447 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.547299 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" event={"ID":"d2de5a71-1d70-4ce5-828d-babfbc31e7a5","Type":"ContainerStarted","Data":"66342065ac736b3f8a19458a919890999781df8a6331dbcae5993179476f2233"} Jul 25 00:19:16.564991 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:16.564944 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" podStartSLOduration=1.5649291779999999 podStartE2EDuration="1.564929178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:19:16.563803918 +0000 UTC m=+3016.256644744" watchObservedRunningTime="2026-07-25 00:19:16.564929178 +0000 UTC m=+3016.257770007" Jul 25 00:19:17.063460 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:17.063426 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:17.068080 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:17.068053 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:17.553325 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:17.553287 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:17.554130 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:17.554108 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:37.670683 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:37.670647 2586 generic.go:356] "Generic (PLEG): container finished" podID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerID="9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207" exitCode=0 Jul 25 00:19:37.671064 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:37.670715 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" event={"ID":"d2de5a71-1d70-4ce5-828d-babfbc31e7a5","Type":"ContainerDied","Data":"9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207"} Jul 25 00:19:37.671130 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:37.671116 2586 scope.go:117] "RemoveContainer" containerID="9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207" Jul 25 00:19:37.672265 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:37.672226 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:19:37.672329 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:37.672313 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:19:37.672365 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:37.672347 2586 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 25 00:19:38.677490 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:38.677458 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" event={"ID":"d2de5a71-1d70-4ce5-828d-babfbc31e7a5","Type":"ContainerStarted","Data":"7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9"} Jul 25 00:19:39.063263 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:39.063139 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:39.063459 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:39.063392 2586 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 25 00:19:39.063548 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:39.063465 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 25 00:19:40.063702 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:40.063667 2586 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 25 00:19:40.064115 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:40.063736 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 25 00:19:41.063510 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:41.063460 2586 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 25 00:19:41.063717 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:41.063542 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 25 00:19:42.064202 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:42.064159 2586 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 25 00:19:42.064720 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:42.064224 2586 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 25 00:19:43.067491 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:43.067455 2586 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:43.067862 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:43.067821 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:19:43.068477 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:19:43.068458 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:20:01.101884 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:01.101810 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl"] Jul 25 00:20:01.102318 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:01.102069 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" containerID="cri-o://7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9" gracePeriod=30 Jul 25 00:20:06.365160 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.365132 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:20:06.478419 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478341 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kj9cv\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-kube-api-access-kj9cv\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478419 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478398 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-credential-socket\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478593 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478439 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istiod-ca-cert\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478593 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478485 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-token\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478593 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478508 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-envoy\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478644 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-socket\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478686 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-podinfo\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478730 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-certs\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478890 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478761 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-data\") pod \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\" (UID: \"d2de5a71-1d70-4ce5-828d-babfbc31e7a5\") " Jul 25 00:20:06.478890 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478768 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:20:06.478993 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478883 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:20:06.478993 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.478904 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:20:06.479170 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.479144 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:20:06.479258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.479178 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.479258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.479201 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.479258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.479215 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.479635 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.479614 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:20:06.480661 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.480638 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:20:06.480796 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.480765 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-kube-api-access-kj9cv" (OuterVolumeSpecName: "kube-api-access-kj9cv") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "kube-api-access-kj9cv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:20:06.481034 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.481017 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:20:06.481093 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.481033 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d2de5a71-1d70-4ce5-828d-babfbc31e7a5" (UID: "d2de5a71-1d70-4ce5-828d-babfbc31e7a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:20:06.579684 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.579658 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj9cv\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-kube-api-access-kj9cv\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.579684 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.579683 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.579828 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.579693 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.579828 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.579701 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.579828 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.579708 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.579828 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.579715 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2de5a71-1d70-4ce5-828d-babfbc31e7a5-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:06.837314 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.837285 2586 generic.go:356] "Generic (PLEG): container finished" podID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerID="7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9" exitCode=0 Jul 25 00:20:06.837314 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.837322 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" event={"ID":"d2de5a71-1d70-4ce5-828d-babfbc31e7a5","Type":"ContainerDied","Data":"7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9"} Jul 25 00:20:06.837523 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.837359 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" event={"ID":"d2de5a71-1d70-4ce5-828d-babfbc31e7a5","Type":"ContainerDied","Data":"66342065ac736b3f8a19458a919890999781df8a6331dbcae5993179476f2233"} Jul 25 00:20:06.837523 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.837381 2586 scope.go:117] "RemoveContainer" containerID="7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9" Jul 25 00:20:06.837523 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.837399 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl" Jul 25 00:20:06.846794 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.846776 2586 scope.go:117] "RemoveContainer" containerID="9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207" Jul 25 00:20:06.854089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.854068 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl"] Jul 25 00:20:06.855418 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.855399 2586 scope.go:117] "RemoveContainer" containerID="7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9" Jul 25 00:20:06.855880 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:20:06.855854 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9\": container with ID starting with 7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9 not found: ID does not exist" containerID="7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9" Jul 25 00:20:06.855983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.855887 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9"} err="failed to get container status \"7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9\": rpc error: code = NotFound desc = could not find container \"7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9\": container with ID starting with 7243512e88188004b4c3503e60c8f8b9a74a05f3e478cb04bea07d71ef1a6ee9 not found: ID does not exist" Jul 25 00:20:06.855983 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.855910 2586 scope.go:117] "RemoveContainer" containerID="9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207" Jul 25 00:20:06.856230 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:20:06.856205 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207\": container with ID starting with 9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207 not found: ID does not exist" containerID="9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207" Jul 25 00:20:06.856300 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.856258 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207"} err="failed to get container status \"9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207\": rpc error: code = NotFound desc = could not find container \"9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207\": container with ID starting with 9c37928845d1f956229633f02980993d8a1887bc0eeb1dff02caadd6dae0a207 not found: ID does not exist" Jul 25 00:20:06.858327 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:06.858308 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-85930d-openshift-default-6d6c65c866-25jhl"] Jul 25 00:20:08.786984 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:08.786953 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" path="/var/lib/kubelet/pods/d2de5a71-1d70-4ce5-828d-babfbc31e7a5/volumes" Jul 25 00:20:24.024322 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.024279 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq"] Jul 25 00:20:24.024801 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.024690 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" podUID="55b4abf6-cf04-42a0-a399-6a0324e240b3" containerName="payload-processing" containerID="cri-o://07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd" gracePeriod=30 Jul 25 00:20:24.306460 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.306433 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:20:24.340968 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.340919 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b4abf6-cf04-42a0-a399-6a0324e240b3-plugins-config-volume\") pod \"55b4abf6-cf04-42a0-a399-6a0324e240b3\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " Jul 25 00:20:24.341321 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.341301 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8qp5\" (UniqueName: \"kubernetes.io/projected/55b4abf6-cf04-42a0-a399-6a0324e240b3-kube-api-access-x8qp5\") pod \"55b4abf6-cf04-42a0-a399-6a0324e240b3\" (UID: \"55b4abf6-cf04-42a0-a399-6a0324e240b3\") " Jul 25 00:20:24.341686 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.341653 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b4abf6-cf04-42a0-a399-6a0324e240b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "55b4abf6-cf04-42a0-a399-6a0324e240b3" (UID: "55b4abf6-cf04-42a0-a399-6a0324e240b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:20:24.341861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.341842 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b4abf6-cf04-42a0-a399-6a0324e240b3-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:24.344388 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.344359 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b4abf6-cf04-42a0-a399-6a0324e240b3-kube-api-access-x8qp5" (OuterVolumeSpecName: "kube-api-access-x8qp5") pod "55b4abf6-cf04-42a0-a399-6a0324e240b3" (UID: "55b4abf6-cf04-42a0-a399-6a0324e240b3"). InnerVolumeSpecName "kube-api-access-x8qp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:20:24.442704 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.442657 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8qp5\" (UniqueName: \"kubernetes.io/projected/55b4abf6-cf04-42a0-a399-6a0324e240b3-kube-api-access-x8qp5\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:20:24.936640 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.936604 2586 generic.go:356] "Generic (PLEG): container finished" podID="55b4abf6-cf04-42a0-a399-6a0324e240b3" containerID="07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd" exitCode=0 Jul 25 00:20:24.936824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.936665 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" Jul 25 00:20:24.936824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.936692 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" event={"ID":"55b4abf6-cf04-42a0-a399-6a0324e240b3","Type":"ContainerDied","Data":"07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd"} Jul 25 00:20:24.936824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.936766 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq" event={"ID":"55b4abf6-cf04-42a0-a399-6a0324e240b3","Type":"ContainerDied","Data":"24e470a15a5cfddb87cc92993a65bf035d1335703b9937371c68b18030d3ae32"} Jul 25 00:20:24.936824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.936797 2586 scope.go:117] "RemoveContainer" containerID="07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd" Jul 25 00:20:24.947176 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.947155 2586 scope.go:117] "RemoveContainer" containerID="07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd" Jul 25 00:20:24.947474 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:20:24.947451 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd\": container with ID starting with 07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd not found: ID does not exist" containerID="07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd" Jul 25 00:20:24.947541 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.947482 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd"} err="failed to get container status \"07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd\": rpc error: code = NotFound desc = could not find container \"07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd\": container with ID starting with 07472a029230bb70e1c74a084492cef54c8e468bbefd6964f5495db34bc608dd not found: ID does not exist" Jul 25 00:20:24.951831 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.951812 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq"] Jul 25 00:20:24.954432 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:24.954412 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-be7457-77f8c868fd-qrmvq"] Jul 25 00:20:26.787137 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:20:26.787093 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55b4abf6-cf04-42a0-a399-6a0324e240b3" path="/var/lib/kubelet/pods/55b4abf6-cf04-42a0-a399-6a0324e240b3/volumes" Jul 25 00:21:13.400411 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:13.400379 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg"] Jul 25 00:21:13.401331 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:13.400626 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" containerID="cri-o://69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7" gracePeriod=30 Jul 25 00:21:18.660191 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.660167 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:21:18.730703 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730643 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-token\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730703 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730701 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-certs\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730721 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istiod-ca-cert\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730740 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-socket\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730764 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-data\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730784 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-podinfo\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730808 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-credential-socket\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.730895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730856 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-envoy\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.731279 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.730922 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pwng\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-kube-api-access-6pwng\") pod \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\" (UID: \"e254f8c6-b61e-4ff5-9dce-40fcdf82613a\") " Jul 25 00:21:18.731279 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731039 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:21:18.731279 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731099 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:21:18.731456 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731270 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:21:18.731456 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731330 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.731456 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731352 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.731456 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731384 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:21:18.731781 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.731753 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-data" (OuterVolumeSpecName: "istio-data") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:21:18.733075 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.733050 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-kube-api-access-6pwng" (OuterVolumeSpecName: "kube-api-access-6pwng") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "kube-api-access-6pwng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:21:18.733172 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.733116 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:21:18.733325 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.733306 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-token" (OuterVolumeSpecName: "istio-token") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:21:18.733391 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.733336 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e254f8c6-b61e-4ff5-9dce-40fcdf82613a" (UID: "e254f8c6-b61e-4ff5-9dce-40fcdf82613a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:21:18.832258 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832207 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pwng\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-kube-api-access-6pwng\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.832375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832265 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.832375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832281 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.832375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832317 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.832375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832330 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.832375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832342 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:18.832375 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:18.832354 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e254f8c6-b61e-4ff5-9dce-40fcdf82613a-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:19.232447 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.232415 2586 generic.go:356] "Generic (PLEG): container finished" podID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerID="69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7" exitCode=0 Jul 25 00:21:19.232638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.232468 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" Jul 25 00:21:19.232638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.232479 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" event={"ID":"e254f8c6-b61e-4ff5-9dce-40fcdf82613a","Type":"ContainerDied","Data":"69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7"} Jul 25 00:21:19.232638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.232527 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg" event={"ID":"e254f8c6-b61e-4ff5-9dce-40fcdf82613a","Type":"ContainerDied","Data":"96efe0adee08ed00562c1f244da344e384e83cc4444e3fdfc6d52aec15c665d9"} Jul 25 00:21:19.232638 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.232552 2586 scope.go:117] "RemoveContainer" containerID="69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7" Jul 25 00:21:19.241618 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.241601 2586 scope.go:117] "RemoveContainer" containerID="cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269" Jul 25 00:21:19.248826 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.248803 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg"] Jul 25 00:21:19.250114 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.250059 2586 scope.go:117] "RemoveContainer" containerID="69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7" Jul 25 00:21:19.250479 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:21:19.250456 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7\": container with ID starting with 69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7 not found: ID does not exist" containerID="69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7" Jul 25 00:21:19.250581 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.250491 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7"} err="failed to get container status \"69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7\": rpc error: code = NotFound desc = could not find container \"69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7\": container with ID starting with 69a09f43f4268f56525d7f4384ddeed7bdb4c29d418f6ccb6a72cf6571fa0db7 not found: ID does not exist" Jul 25 00:21:19.250581 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.250512 2586 scope.go:117] "RemoveContainer" containerID="cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269" Jul 25 00:21:19.250805 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:21:19.250787 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269\": container with ID starting with cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269 not found: ID does not exist" containerID="cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269" Jul 25 00:21:19.250869 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.250811 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269"} err="failed to get container status \"cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269\": rpc error: code = NotFound desc = could not find container \"cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269\": container with ID starting with cc8e0bb983bd141e2ba43a12148ea013546125ab010566ba93ec3a016f157269 not found: ID does not exist" Jul 25 00:21:19.252413 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:19.252392 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-be7457-openshift-default-7c954c5ffd-lcsqg"] Jul 25 00:21:20.786895 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:20.786849 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" path="/var/lib/kubelet/pods/e254f8c6-b61e-4ff5-9dce-40fcdf82613a/volumes" Jul 25 00:21:24.319820 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.319740 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh"] Jul 25 00:21:24.320402 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.319987 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" podUID="4ab8d029-0673-4620-9c0b-a72b92d186c1" containerName="payload-pre-processing" containerID="cri-o://0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b" gracePeriod=30 Jul 25 00:21:24.580053 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.579972 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:21:24.678572 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.678536 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8d029-0673-4620-9c0b-a72b92d186c1-plugins-config-volume\") pod \"4ab8d029-0673-4620-9c0b-a72b92d186c1\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " Jul 25 00:21:24.678749 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.678605 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz7nc\" (UniqueName: \"kubernetes.io/projected/4ab8d029-0673-4620-9c0b-a72b92d186c1-kube-api-access-fz7nc\") pod \"4ab8d029-0673-4620-9c0b-a72b92d186c1\" (UID: \"4ab8d029-0673-4620-9c0b-a72b92d186c1\") " Jul 25 00:21:24.679021 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.678994 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab8d029-0673-4620-9c0b-a72b92d186c1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4ab8d029-0673-4620-9c0b-a72b92d186c1" (UID: "4ab8d029-0673-4620-9c0b-a72b92d186c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:21:24.680747 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.680724 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab8d029-0673-4620-9c0b-a72b92d186c1-kube-api-access-fz7nc" (OuterVolumeSpecName: "kube-api-access-fz7nc") pod "4ab8d029-0673-4620-9c0b-a72b92d186c1" (UID: "4ab8d029-0673-4620-9c0b-a72b92d186c1"). InnerVolumeSpecName "kube-api-access-fz7nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:21:24.779308 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.779268 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ab8d029-0673-4620-9c0b-a72b92d186c1-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:24.779308 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:24.779302 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz7nc\" (UniqueName: \"kubernetes.io/projected/4ab8d029-0673-4620-9c0b-a72b92d186c1-kube-api-access-fz7nc\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:21:25.269058 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.269016 2586 generic.go:356] "Generic (PLEG): container finished" podID="4ab8d029-0673-4620-9c0b-a72b92d186c1" containerID="0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b" exitCode=0 Jul 25 00:21:25.269307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.269070 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" Jul 25 00:21:25.269307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.269115 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" event={"ID":"4ab8d029-0673-4620-9c0b-a72b92d186c1","Type":"ContainerDied","Data":"0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b"} Jul 25 00:21:25.269307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.269186 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh" event={"ID":"4ab8d029-0673-4620-9c0b-a72b92d186c1","Type":"ContainerDied","Data":"12dacc4751527d7da8a854c7171ab5fb916bab5089b630d2c08591309c180b9b"} Jul 25 00:21:25.269307 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.269220 2586 scope.go:117] "RemoveContainer" containerID="0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b" Jul 25 00:21:25.280787 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.280760 2586 scope.go:117] "RemoveContainer" containerID="0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b" Jul 25 00:21:25.281063 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:21:25.281042 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b\": container with ID starting with 0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b not found: ID does not exist" containerID="0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b" Jul 25 00:21:25.281127 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.281074 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b"} err="failed to get container status \"0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b\": rpc error: code = NotFound desc = could not find container \"0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b\": container with ID starting with 0278ca5f7629b845c32bbf992955f832c56ae374597b261e9bc749fa2f913c8b not found: ID does not exist" Jul 25 00:21:25.292573 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.292541 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh"] Jul 25 00:21:25.296086 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:25.296058 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-899559-79ddbd9648-dtwkh"] Jul 25 00:21:26.787314 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:26.787281 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ab8d029-0673-4620-9c0b-a72b92d186c1" path="/var/lib/kubelet/pods/4ab8d029-0673-4620-9c0b-a72b92d186c1/volumes" Jul 25 00:21:57.510089 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:57.510048 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7"] Jul 25 00:21:57.510692 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:21:57.510314 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" containerID="cri-o://5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33" gracePeriod=30 Jul 25 00:22:02.768475 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.768449 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:22:02.919126 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919091 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z97xs\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-kube-api-access-z97xs\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919172 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-token\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919194 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701381fd-057b-4f2b-be78-d874d5d0deaa-istiod-ca-cert\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919218 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-data\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919266 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-credential-socket\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919285 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-envoy\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919350 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-podinfo\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919386 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-socket\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919444 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919408 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-certs\") pod \"701381fd-057b-4f2b-be78-d874d5d0deaa\" (UID: \"701381fd-057b-4f2b-be78-d874d5d0deaa\") " Jul 25 00:22:02.919869 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919711 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/701381fd-057b-4f2b-be78-d874d5d0deaa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:22:02.919869 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919830 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:02.919976 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.919919 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:02.920087 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.920068 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:02.920526 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.920500 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-data" (OuterVolumeSpecName: "istio-data") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:02.921649 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.921615 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:22:02.921649 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.921638 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-token" (OuterVolumeSpecName: "istio-token") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:22:02.921649 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.921622 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-kube-api-access-z97xs" (OuterVolumeSpecName: "kube-api-access-z97xs") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "kube-api-access-z97xs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:22:02.921869 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:02.921833 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "701381fd-057b-4f2b-be78-d874d5d0deaa" (UID: "701381fd-057b-4f2b-be78-d874d5d0deaa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:03.021055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021025 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021049 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021058 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021067 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z97xs\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-kube-api-access-z97xs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021076 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021085 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/701381fd-057b-4f2b-be78-d874d5d0deaa-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021092 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021100 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.021316 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.021107 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/701381fd-057b-4f2b-be78-d874d5d0deaa-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:03.471649 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.471616 2586 generic.go:356] "Generic (PLEG): container finished" podID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerID="5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33" exitCode=0 Jul 25 00:22:03.471819 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.471668 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" Jul 25 00:22:03.471819 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.471695 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" event={"ID":"701381fd-057b-4f2b-be78-d874d5d0deaa","Type":"ContainerDied","Data":"5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33"} Jul 25 00:22:03.471819 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.471745 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7" event={"ID":"701381fd-057b-4f2b-be78-d874d5d0deaa","Type":"ContainerDied","Data":"b0313d09029ba9dcaf239f95c0d840fe0e984e7a73a7ec5e805758973744b092"} Jul 25 00:22:03.471819 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.471770 2586 scope.go:117] "RemoveContainer" containerID="5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33" Jul 25 00:22:03.481311 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.481211 2586 scope.go:117] "RemoveContainer" containerID="0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179" Jul 25 00:22:03.489320 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.489304 2586 scope.go:117] "RemoveContainer" containerID="5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33" Jul 25 00:22:03.489557 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:22:03.489537 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33\": container with ID starting with 5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33 not found: ID does not exist" containerID="5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33" Jul 25 00:22:03.489632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.489563 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33"} err="failed to get container status \"5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33\": rpc error: code = NotFound desc = could not find container \"5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33\": container with ID starting with 5f81e6fd21210ebd92af40e250a58d4d15f140f0780f540d4b24fe8535925c33 not found: ID does not exist" Jul 25 00:22:03.489632 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.489579 2586 scope.go:117] "RemoveContainer" containerID="0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179" Jul 25 00:22:03.489774 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:22:03.489760 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179\": container with ID starting with 0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179 not found: ID does not exist" containerID="0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179" Jul 25 00:22:03.489813 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.489777 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179"} err="failed to get container status \"0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179\": rpc error: code = NotFound desc = could not find container \"0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179\": container with ID starting with 0380bfb4545e2af0b43ab7fa816dd42f4310acee44deb072ebc7c7f80d43d179 not found: ID does not exist" Jul 25 00:22:03.492625 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.492600 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7"] Jul 25 00:22:03.496373 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:03.496352 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-92df89-openshift-default-85c894fd84-n4mj7"] Jul 25 00:22:04.786870 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:04.786840 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" path="/var/lib/kubelet/pods/701381fd-057b-4f2b-be78-d874d5d0deaa/volumes" Jul 25 00:22:08.998745 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:08.998710 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km"] Jul 25 00:22:08.999723 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:08.999530 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" podUID="24801cd3-8cee-4fc4-a00e-02f9cf628e13" containerName="payload-processing" containerID="cri-o://d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9" gracePeriod=30 Jul 25 00:22:09.001705 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.001683 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9"] Jul 25 00:22:09.001903 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.001881 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" podUID="b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" containerName="payload-pre-processing" containerID="cri-o://ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710" gracePeriod=30 Jul 25 00:22:09.266431 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.266407 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:22:09.297487 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.297466 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:22:09.371348 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.371325 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-plugins-config-volume\") pod \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " Jul 25 00:22:09.371480 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.371360 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pqm4t\" (UniqueName: \"kubernetes.io/projected/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-kube-api-access-pqm4t\") pod \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\" (UID: \"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62\") " Jul 25 00:22:09.371671 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.371648 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" (UID: "b5ef2a5a-c1fe-4217-81fb-36304f9f0f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:22:09.373382 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.373362 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-kube-api-access-pqm4t" (OuterVolumeSpecName: "kube-api-access-pqm4t") pod "b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" (UID: "b5ef2a5a-c1fe-4217-81fb-36304f9f0f62"). InnerVolumeSpecName "kube-api-access-pqm4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:22:09.471965 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.471939 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nntvm\" (UniqueName: \"kubernetes.io/projected/24801cd3-8cee-4fc4-a00e-02f9cf628e13-kube-api-access-nntvm\") pod \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " Jul 25 00:22:09.472113 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.472099 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24801cd3-8cee-4fc4-a00e-02f9cf628e13-plugins-config-volume\") pod \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\" (UID: \"24801cd3-8cee-4fc4-a00e-02f9cf628e13\") " Jul 25 00:22:09.472407 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.472389 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:09.472466 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.472411 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqm4t\" (UniqueName: \"kubernetes.io/projected/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62-kube-api-access-pqm4t\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:09.472512 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.472458 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24801cd3-8cee-4fc4-a00e-02f9cf628e13-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "24801cd3-8cee-4fc4-a00e-02f9cf628e13" (UID: "24801cd3-8cee-4fc4-a00e-02f9cf628e13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:22:09.473940 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.473923 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24801cd3-8cee-4fc4-a00e-02f9cf628e13-kube-api-access-nntvm" (OuterVolumeSpecName: "kube-api-access-nntvm") pod "24801cd3-8cee-4fc4-a00e-02f9cf628e13" (UID: "24801cd3-8cee-4fc4-a00e-02f9cf628e13"). InnerVolumeSpecName "kube-api-access-nntvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:22:09.504048 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.503965 2586 generic.go:356] "Generic (PLEG): container finished" podID="24801cd3-8cee-4fc4-a00e-02f9cf628e13" containerID="d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9" exitCode=0 Jul 25 00:22:09.504048 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.504010 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" Jul 25 00:22:09.504189 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.504063 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" event={"ID":"24801cd3-8cee-4fc4-a00e-02f9cf628e13","Type":"ContainerDied","Data":"d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9"} Jul 25 00:22:09.504189 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.504134 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km" event={"ID":"24801cd3-8cee-4fc4-a00e-02f9cf628e13","Type":"ContainerDied","Data":"1ccb173dbbac7ef9075313e21e6bbe7d916d7482f9500016a7120ab5903b533f"} Jul 25 00:22:09.504189 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.504151 2586 scope.go:117] "RemoveContainer" containerID="d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9" Jul 25 00:22:09.505525 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.505501 2586 generic.go:356] "Generic (PLEG): container finished" podID="b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" containerID="ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710" exitCode=0 Jul 25 00:22:09.505626 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.505535 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" Jul 25 00:22:09.505626 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.505588 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" event={"ID":"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62","Type":"ContainerDied","Data":"ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710"} Jul 25 00:22:09.505735 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.505630 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9" event={"ID":"b5ef2a5a-c1fe-4217-81fb-36304f9f0f62","Type":"ContainerDied","Data":"8f0786151cedc649f7315b0d41a48325911ebe5e96bfeae8837230f0ee8a3d0b"} Jul 25 00:22:09.514475 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.514455 2586 scope.go:117] "RemoveContainer" containerID="d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9" Jul 25 00:22:09.514698 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:22:09.514681 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9\": container with ID starting with d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9 not found: ID does not exist" containerID="d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9" Jul 25 00:22:09.514755 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.514704 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9"} err="failed to get container status \"d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9\": rpc error: code = NotFound desc = could not find container \"d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9\": container with ID starting with d88edff95cc98d1676777a17b67a3b541cc10a39788d743f4663873202cde8b9 not found: ID does not exist" Jul 25 00:22:09.514755 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.514720 2586 scope.go:117] "RemoveContainer" containerID="ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710" Jul 25 00:22:09.522672 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.522650 2586 scope.go:117] "RemoveContainer" containerID="ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710" Jul 25 00:22:09.522942 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:22:09.522916 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710\": container with ID starting with ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710 not found: ID does not exist" containerID="ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710" Jul 25 00:22:09.523031 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.522948 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710"} err="failed to get container status \"ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710\": rpc error: code = NotFound desc = could not find container \"ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710\": container with ID starting with ab4f484a1951593771b22fbcaab6ec0b1260c03c0c2e33b3dd673d284c434710 not found: ID does not exist" Jul 25 00:22:09.525073 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.525052 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km"] Jul 25 00:22:09.529415 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.529379 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-413844-74879c78c8-qm4km"] Jul 25 00:22:09.538352 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.538327 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9"] Jul 25 00:22:09.542859 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.542833 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-413844-5748887b4-dgzf9"] Jul 25 00:22:09.572795 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.572770 2586 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24801cd3-8cee-4fc4-a00e-02f9cf628e13-plugins-config-volume\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:09.572795 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:09.572793 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntvm\" (UniqueName: \"kubernetes.io/projected/24801cd3-8cee-4fc4-a00e-02f9cf628e13-kube-api-access-nntvm\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:10.787929 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:10.787888 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24801cd3-8cee-4fc4-a00e-02f9cf628e13" path="/var/lib/kubelet/pods/24801cd3-8cee-4fc4-a00e-02f9cf628e13/volumes" Jul 25 00:22:10.788460 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:10.788431 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" path="/var/lib/kubelet/pods/b5ef2a5a-c1fe-4217-81fb-36304f9f0f62/volumes" Jul 25 00:22:19.715960 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:19.715925 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq"] Jul 25 00:22:19.716473 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:19.716303 2586 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" containerID="cri-o://15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2" gracePeriod=30 Jul 25 00:22:24.988331 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:24.988260 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:22:25.110501 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110463 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-credential-socket\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110514 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-data\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110535 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-certs\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110572 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-kube-api-access-bxt4g\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110590 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-envoy\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110615 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-socket\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110640 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-podinfo\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.110710 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110661 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-token\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.111082 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110762 2586 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istiod-ca-cert\") pod \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\" (UID: \"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5\") " Jul 25 00:22:25.111082 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.110798 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:25.111082 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.111004 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:25.111082 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.111074 2586 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-credential-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.111337 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.111223 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:25.111405 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.111319 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 25 00:22:25.111938 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.111906 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-data" (OuterVolumeSpecName: "istio-data") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:25.113139 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.113113 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 25 00:22:25.113265 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.113127 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 25 00:22:25.113265 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.113139 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-token" (OuterVolumeSpecName: "istio-token") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:22:25.113265 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.113153 2586 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-kube-api-access-bxt4g" (OuterVolumeSpecName: "kube-api-access-bxt4g") pod "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" (UID: "91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5"). InnerVolumeSpecName "kube-api-access-bxt4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 25 00:22:25.212302 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212257 2586 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-data\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212302 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212293 2586 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-certs\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212302 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212303 2586 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxt4g\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-kube-api-access-bxt4g\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212302 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212312 2586 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-envoy\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212592 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212323 2586 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-workload-socket\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212592 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212335 2586 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-podinfo\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212592 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212344 2586 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istio-token\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.212592 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.212352 2586 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5-istiod-ca-cert\") on node \"ip-10-0-131-63.ec2.internal\" DevicePath \"\"" Jul 25 00:22:25.592847 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.592809 2586 generic.go:356] "Generic (PLEG): container finished" podID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerID="15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2" exitCode=0 Jul 25 00:22:25.593055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.592871 2586 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" Jul 25 00:22:25.593055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.592883 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" event={"ID":"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5","Type":"ContainerDied","Data":"15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2"} Jul 25 00:22:25.593055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.592952 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq" event={"ID":"91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5","Type":"ContainerDied","Data":"2bf7c2ff5b4471da25342d06f264119aa7a1e737a3c3822e52c2d6b935c5912a"} Jul 25 00:22:25.593055 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.592974 2586 scope.go:117] "RemoveContainer" containerID="15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2" Jul 25 00:22:25.603349 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.603321 2586 scope.go:117] "RemoveContainer" containerID="4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211" Jul 25 00:22:25.613457 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.613347 2586 scope.go:117] "RemoveContainer" containerID="15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2" Jul 25 00:22:25.613712 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:22:25.613683 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2\": container with ID starting with 15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2 not found: ID does not exist" containerID="15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2" Jul 25 00:22:25.613869 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.613724 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2"} err="failed to get container status \"15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2\": rpc error: code = NotFound desc = could not find container \"15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2\": container with ID starting with 15806d9173ff948fcaa6c3b9e038520549404eb7506fe114aa8ab34ac53405f2 not found: ID does not exist" Jul 25 00:22:25.613869 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.613748 2586 scope.go:117] "RemoveContainer" containerID="4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211" Jul 25 00:22:25.614101 ip-10-0-131-63 kubenswrapper[2586]: E0725 00:22:25.614076 2586 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211\": container with ID starting with 4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211 not found: ID does not exist" containerID="4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211" Jul 25 00:22:25.614164 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.614113 2586 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211"} err="failed to get container status \"4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211\": rpc error: code = NotFound desc = could not find container \"4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211\": container with ID starting with 4307109bd450b1d7eee7fb9291276f66600cee76419866d72a5127b5a7e48211 not found: ID does not exist" Jul 25 00:22:25.615980 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.615960 2586 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq"] Jul 25 00:22:25.619805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:25.619785 2586 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-413844-openshift-default-67766bf844-97tzq"] Jul 25 00:22:26.329463 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:26.329432 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-qksm2_8b8cb50c-23ae-404d-8e75-95b7bd9ca504/manager/0.log" Jul 25 00:22:26.454889 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:26.454856 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sn8x8_6ddfaabf-761b-4973-a628-64aed03c455b/manager/0.log" Jul 25 00:22:26.713706 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:26.713679 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dx2nm_de02cba9-3c04-4368-8307-27ee04bff695/server/0.log" Jul 25 00:22:26.786494 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:26.786459 2586 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" path="/var/lib/kubelet/pods/91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5/volumes" Jul 25 00:22:27.258954 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:27.258923 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69778f9898-vxr7r_a3576e52-c673-44cf-a992-46b94db07d5a/copy-manifests/0.log" Jul 25 00:22:27.271404 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:27.271376 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69778f9898-vxr7r_a3576e52-c673-44cf-a992-46b94db07d5a/manager/0.log" Jul 25 00:22:27.784396 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:27.784366 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748945-rb5bn_53d03129-4ee9-468b-847f-3dc95de27a0d/cleanup/0.log" Jul 25 00:22:29.678484 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:29.678452 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7rx8p_ecbc90ba-0969-41a6-8517-aeb74095097e/manager/0.log" Jul 25 00:22:29.799739 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:29.799709 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5bxmm_d08a8ae1-982c-4a7f-9629-cfecc6d58f87/manager/0.log" Jul 25 00:22:30.042771 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:30.042693 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wjwlw_0ba1902c-ecd9-4a9f-b87a-9fc90aa74390/registry-server/0.log" Jul 25 00:22:30.174214 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:30.174152 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8zm98_dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a/manager/0.log" Jul 25 00:22:30.431861 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:30.431824 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-fbt6p_79060740-4a78-4bfe-a642-2271705f8576/manager/0.log" Jul 25 00:22:31.219478 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:31.219447 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d7d7d465-f2xmv_662651c3-c530-477e-9071-4aa47747f54d/kube-auth-proxy/0.log" Jul 25 00:22:31.488418 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:31.488337 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dkr6m_0dbfa169-1d5f-492b-b938-94f02db53897/payload-pre-processing/0.log" Jul 25 00:22:31.742961 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:31.742864 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f648499-fvcgf_d363ed1b-691f-4e1e-9416-5a8bb29e0a43/router/0.log" Jul 25 00:22:32.116332 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:32.116304 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-2d2pt_7fc943ed-3253-4673-8702-062707fde29f/main/0.log" Jul 25 00:22:32.366524 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:32.366445 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-zgbql_76a05693-1be0-4b6d-9969-dc78215c15b4/main/0.log" Jul 25 00:22:38.722323 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722283 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t745v/must-gather-khls8"] Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722671 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24801cd3-8cee-4fc4-a00e-02f9cf628e13" containerName="payload-processing" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722683 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="24801cd3-8cee-4fc4-a00e-02f9cf628e13" containerName="payload-processing" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722696 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55b4abf6-cf04-42a0-a399-6a0324e240b3" containerName="payload-processing" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722702 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="55b4abf6-cf04-42a0-a399-6a0324e240b3" containerName="payload-processing" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722709 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722715 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722722 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722727 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722741 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722746 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722752 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ab8d029-0673-4620-9c0b-a72b92d186c1" containerName="payload-pre-processing" Jul 25 00:22:38.722751 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722757 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ab8d029-0673-4620-9c0b-a72b92d186c1" containerName="payload-pre-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722764 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722770 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722778 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722784 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722790 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" containerName="payload-pre-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722795 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" containerName="payload-pre-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722802 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722807 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722815 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722820 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722877 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="24801cd3-8cee-4fc4-a00e-02f9cf628e13" containerName="payload-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722885 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5ef2a5a-c1fe-4217-81fb-36304f9f0f62" containerName="payload-pre-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722891 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722896 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722901 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="e254f8c6-b61e-4ff5-9dce-40fcdf82613a" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722907 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722912 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="91f6709f-5b1c-44e6-85e3-b9ebbcc0a5c5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722916 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722923 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ab8d029-0673-4620-9c0b-a72b92d186c1" containerName="payload-pre-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722928 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="55b4abf6-cf04-42a0-a399-6a0324e240b3" containerName="payload-processing" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722934 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2de5a71-1d70-4ce5-828d-babfbc31e7a5" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722982 2586 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.722987 2586 state_mem.go:107] "Deleted CPUSet assignment" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" Jul 25 00:22:38.723119 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.723053 2586 memory_manager.go:368] "RemoveStaleState removing state" podUID="701381fd-057b-4f2b-be78-d874d5d0deaa" containerName="istio-proxy" Jul 25 00:22:38.727824 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.727808 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:38.730411 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.730389 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t745v\"/\"openshift-service-ca.crt\"" Jul 25 00:22:38.730411 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.730406 2586 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-t745v\"/\"kube-root-ca.crt\"" Jul 25 00:22:38.731197 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.731177 2586 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-t745v\"/\"default-dockercfg-f6z68\"" Jul 25 00:22:38.734838 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.734813 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t745v/must-gather-khls8"] Jul 25 00:22:38.835623 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.835581 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/649ecd7a-ebc9-4016-91aa-6986619464f4-must-gather-output\") pod \"must-gather-khls8\" (UID: \"649ecd7a-ebc9-4016-91aa-6986619464f4\") " pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:38.835805 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.835721 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khvdn\" (UniqueName: \"kubernetes.io/projected/649ecd7a-ebc9-4016-91aa-6986619464f4-kube-api-access-khvdn\") pod \"must-gather-khls8\" (UID: \"649ecd7a-ebc9-4016-91aa-6986619464f4\") " pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:38.936561 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.936519 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khvdn\" (UniqueName: \"kubernetes.io/projected/649ecd7a-ebc9-4016-91aa-6986619464f4-kube-api-access-khvdn\") pod \"must-gather-khls8\" (UID: \"649ecd7a-ebc9-4016-91aa-6986619464f4\") " pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:38.936706 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.936592 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/649ecd7a-ebc9-4016-91aa-6986619464f4-must-gather-output\") pod \"must-gather-khls8\" (UID: \"649ecd7a-ebc9-4016-91aa-6986619464f4\") " pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:38.936939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.936920 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/649ecd7a-ebc9-4016-91aa-6986619464f4-must-gather-output\") pod \"must-gather-khls8\" (UID: \"649ecd7a-ebc9-4016-91aa-6986619464f4\") " pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:38.945084 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:38.945055 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khvdn\" (UniqueName: \"kubernetes.io/projected/649ecd7a-ebc9-4016-91aa-6986619464f4-kube-api-access-khvdn\") pod \"must-gather-khls8\" (UID: \"649ecd7a-ebc9-4016-91aa-6986619464f4\") " pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:39.038214 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:39.038107 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t745v/must-gather-khls8" Jul 25 00:22:39.176397 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:39.176372 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t745v/must-gather-khls8"] Jul 25 00:22:39.178872 ip-10-0-131-63 kubenswrapper[2586]: W0725 00:22:39.178844 2586 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod649ecd7a_ebc9_4016_91aa_6986619464f4.slice/crio-47a1ee86e8af87455c7f8cef91a340786c063c6309e32e6022d3d40272fa6d1e WatchSource:0}: Error finding container 47a1ee86e8af87455c7f8cef91a340786c063c6309e32e6022d3d40272fa6d1e: Status 404 returned error can't find the container with id 47a1ee86e8af87455c7f8cef91a340786c063c6309e32e6022d3d40272fa6d1e Jul 25 00:22:39.180781 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:39.180761 2586 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 25 00:22:39.664939 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:39.664901 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t745v/must-gather-khls8" event={"ID":"649ecd7a-ebc9-4016-91aa-6986619464f4","Type":"ContainerStarted","Data":"47a1ee86e8af87455c7f8cef91a340786c063c6309e32e6022d3d40272fa6d1e"} Jul 25 00:22:40.676546 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:40.676491 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t745v/must-gather-khls8" event={"ID":"649ecd7a-ebc9-4016-91aa-6986619464f4","Type":"ContainerStarted","Data":"2cd2b84e3c7fe5f11eb60abec7f057a377398db5b55f95b1910d06343cc7f255"} Jul 25 00:22:40.677039 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:40.676567 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t745v/must-gather-khls8" event={"ID":"649ecd7a-ebc9-4016-91aa-6986619464f4","Type":"ContainerStarted","Data":"f05726b4332270a08082590b23b99dbf854321e8a278197968faaf426362bc03"} Jul 25 00:22:40.691863 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:40.691817 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t745v/must-gather-khls8" podStartSLOduration=1.845922341 podStartE2EDuration="2.691777612s" totalImagesPullingTime="845.855271ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-25 00:22:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:22:40.690730116 +0000 UTC m=+3220.383570944" watchObservedRunningTime="2026-07-25 00:22:40.691777612 +0000 UTC m=+3220.384618473" Jul 25 00:22:41.514670 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:41.514622 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-h9j6b_5b2c2b1c-c32a-45e8-ac58-b00ff25ffe52/global-pull-secret-syncer/0.log" Jul 25 00:22:41.730310 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:41.730272 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-tj4jt_af3fce95-c009-46a3-81bb-6f841e5c3e6d/konnectivity-agent/0.log" Jul 25 00:22:41.752268 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:41.752218 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-63.ec2.internal_a87cd2889e81c5df0b8771970533f7d1/haproxy/0.log" Jul 25 00:22:47.226986 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:47.226952 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-7rx8p_ecbc90ba-0969-41a6-8517-aeb74095097e/manager/0.log" Jul 25 00:22:47.250254 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:47.250192 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5bxmm_d08a8ae1-982c-4a7f-9629-cfecc6d58f87/manager/0.log" Jul 25 00:22:47.316653 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:47.316623 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-wjwlw_0ba1902c-ecd9-4a9f-b87a-9fc90aa74390/registry-server/0.log" Jul 25 00:22:47.502689 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:47.502605 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-8zm98_dc328c13-43c0-4a82-8d2e-f15f0d2c5c7a/manager/0.log" Jul 25 00:22:47.623919 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:47.623882 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-fbt6p_79060740-4a78-4bfe-a642-2271705f8576/manager/0.log" Jul 25 00:22:49.463733 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:49.463696 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-v4jld_477c6c0e-fc47-4c72-adbe-f697c23204ac/cluster-monitoring-operator/0.log" Jul 25 00:22:49.673775 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:49.673745 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh72p_a422bd0e-5604-4ff7-93c0-01b945fe2a9c/node-exporter/0.log" Jul 25 00:22:49.692908 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:49.692834 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh72p_a422bd0e-5604-4ff7-93c0-01b945fe2a9c/kube-rbac-proxy/0.log" Jul 25 00:22:49.716294 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:49.716197 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mh72p_a422bd0e-5604-4ff7-93c0-01b945fe2a9c/init-textfile/0.log" Jul 25 00:22:51.492588 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.492553 2586 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw"] Jul 25 00:22:51.498466 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.498442 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.505873 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.505508 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw"] Jul 25 00:22:51.519681 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.519658 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-dbg74_f4087967-601a-41de-b506-9a66be19ce38/networking-console-plugin/0.log" Jul 25 00:22:51.667483 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.667448 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-sys\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.667679 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.667628 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-podres\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.667775 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.667745 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-lib-modules\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.667851 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.667796 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62m77\" (UniqueName: \"kubernetes.io/projected/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-kube-api-access-62m77\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.667914 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.667852 2586 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-proc\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.770283 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.770169 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-lib-modules\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.770549 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.770527 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62m77\" (UniqueName: \"kubernetes.io/projected/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-kube-api-access-62m77\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.771198 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.771176 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-proc\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.771522 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.771505 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-sys\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.771839 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.771713 2586 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-podres\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.771997 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.771414 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-proc\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.772101 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.771649 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-sys\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.772191 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.770761 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-lib-modules\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.772285 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.771819 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-podres\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.779848 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.779817 2586 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62m77\" (UniqueName: \"kubernetes.io/projected/8216e4ac-f26b-4fc2-9d4f-74ecaff2e663-kube-api-access-62m77\") pod \"perf-node-gather-daemonset-nf4cw\" (UID: \"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663\") " pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:51.812133 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:51.812095 2586 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:52.016112 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.016016 2586 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw"] Jul 25 00:22:52.175572 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.175516 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/1.log" Jul 25 00:22:52.183953 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.183914 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-mg8tv_233cbdf6-cc00-4133-8011-6b051a2439df/console-operator/2.log" Jul 25 00:22:52.659125 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.659094 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-8rpg2_1582bf21-f4b1-4983-940a-76597189c439/download-server/0.log" Jul 25 00:22:52.790068 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.790037 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:52.790293 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.790101 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" event={"ID":"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663","Type":"ContainerStarted","Data":"7809f02c8b3fe8a27bfa0393a5356eea8a5882570d045c989174e9c206447f55"} Jul 25 00:22:52.790293 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.790129 2586 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" event={"ID":"8216e4ac-f26b-4fc2-9d4f-74ecaff2e663","Type":"ContainerStarted","Data":"a314f173ab776bf778d7fdd773b8a48bb1bcdfa6d2610a3199ad63592edb6dbf"} Jul 25 00:22:52.800473 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:52.800425 2586 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" podStartSLOduration=1.800408954 podStartE2EDuration="1.800408954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-25 00:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-25 00:22:52.799662344 +0000 UTC m=+3232.492503173" watchObservedRunningTime="2026-07-25 00:22:52.800408954 +0000 UTC m=+3232.493249782" Jul 25 00:22:53.234662 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:53.234583 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-p66mv_b43d98b3-bdfd-4517-b631-e4a2292c3b8f/volume-data-source-validator/0.log" Jul 25 00:22:54.302508 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:54.302478 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p54z5_32fd7b44-4b9c-4733-a471-3316a037d9f9/dns/0.log" Jul 25 00:22:54.321068 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:54.321039 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p54z5_32fd7b44-4b9c-4733-a471-3316a037d9f9/kube-rbac-proxy/0.log" Jul 25 00:22:54.341987 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:54.341951 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4rjbq_6ceefe7e-2625-4162-b792-702f2eda86c8/dns-node-resolver/0.log" Jul 25 00:22:54.880931 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:54.880904 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29748960-5kpcj_13c6f888-1063-4bb9-9f8a-0bfd0f1223d2/image-pruner/0.log" Jul 25 00:22:55.003893 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:55.003863 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-zmcf8_0afabe2f-09c6-4e92-9bcf-b233c9d64aa0/node-ca/0.log" Jul 25 00:22:56.225529 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:56.225494 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d7d7d465-f2xmv_662651c3-c530-477e-9071-4aa47747f54d/kube-auth-proxy/0.log" Jul 25 00:22:56.344867 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:56.344836 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dkr6m_0dbfa169-1d5f-492b-b938-94f02db53897/payload-pre-processing/0.log" Jul 25 00:22:56.403592 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:56.403563 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f648499-fvcgf_d363ed1b-691f-4e1e-9416-5a8bb29e0a43/router/0.log" Jul 25 00:22:57.227894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:57.227857 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mxztk_a9a7294c-8419-4d34-bb70-554909333f7e/serve-healthcheck-canary/0.log" Jul 25 00:22:57.683984 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:57.683947 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-8lhwc_c76723fb-9238-4421-bc73-5bf67c9604d6/insights-operator/0.log" Jul 25 00:22:57.685129 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:57.685106 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-8lhwc_c76723fb-9238-4421-bc73-5bf67c9604d6/insights-operator/1.log" Jul 25 00:22:57.707168 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:57.707132 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8m8fd_daba6781-d561-4345-a105-3f0f93979fe5/kube-rbac-proxy/0.log" Jul 25 00:22:57.725714 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:57.725682 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8m8fd_daba6781-d561-4345-a105-3f0f93979fe5/exporter/0.log" Jul 25 00:22:57.745385 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:57.745354 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8m8fd_daba6781-d561-4345-a105-3f0f93979fe5/extractor/0.log" Jul 25 00:22:58.801112 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:58.801086 2586 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-t745v/perf-node-gather-daemonset-nf4cw" Jul 25 00:22:59.895953 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:22:59.895925 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-qksm2_8b8cb50c-23ae-404d-8e75-95b7bd9ca504/manager/0.log" Jul 25 00:23:00.223272 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:00.223228 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-sn8x8_6ddfaabf-761b-4973-a628-64aed03c455b/manager/0.log" Jul 25 00:23:00.697922 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:00.697896 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dx2nm_de02cba9-3c04-4368-8307-27ee04bff695/server/0.log" Jul 25 00:23:01.028292 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:01.028180 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69778f9898-vxr7r_a3576e52-c673-44cf-a992-46b94db07d5a/manager/0.log" Jul 25 00:23:01.044540 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:01.044514 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-69778f9898-vxr7r_a3576e52-c673-44cf-a992-46b94db07d5a/copy-manifests/0.log" Jul 25 00:23:02.472095 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:02.472060 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d7d7d465-f2xmv_662651c3-c530-477e-9071-4aa47747f54d/kube-auth-proxy/0.log" Jul 25 00:23:02.568942 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:02.568901 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dkr6m_0dbfa169-1d5f-492b-b938-94f02db53897/payload-pre-processing/0.log" Jul 25 00:23:02.617915 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:02.617876 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f648499-fvcgf_d363ed1b-691f-4e1e-9416-5a8bb29e0a43/router/0.log" Jul 25 00:23:04.235291 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:04.235262 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6d7d7d465-f2xmv_662651c3-c530-477e-9071-4aa47747f54d/kube-auth-proxy/0.log" Jul 25 00:23:04.339720 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:04.339683 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-dkr6m_0dbfa169-1d5f-492b-b938-94f02db53897/payload-pre-processing/0.log" Jul 25 00:23:04.390031 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:04.390001 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-59f648499-fvcgf_d363ed1b-691f-4e1e-9416-5a8bb29e0a43/router/0.log" Jul 25 00:23:07.165575 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:07.165540 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-8j876_b639319f-b24f-48c6-abcc-06235bcd00c7/migrator/0.log" Jul 25 00:23:07.187295 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:07.187255 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-8j876_b639319f-b24f-48c6-abcc-06235bcd00c7/graceful-termination/0.log" Jul 25 00:23:07.515875 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:07.515792 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88_e8b72a9a-44f2-4d63-866b-62cccd430125/kube-storage-version-migrator-operator/1.log" Jul 25 00:23:07.516658 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:07.516635 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-8fk88_e8b72a9a-44f2-4d63-866b-62cccd430125/kube-storage-version-migrator-operator/0.log" Jul 25 00:23:08.788759 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.788674 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/kube-multus-additional-cni-plugins/0.log" Jul 25 00:23:08.807894 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.807794 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/egress-router-binary-copy/0.log" Jul 25 00:23:08.826833 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.826789 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/cni-plugins/0.log" Jul 25 00:23:08.848196 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.848162 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/bond-cni-plugin/0.log" Jul 25 00:23:08.867392 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.867360 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/routeoverride-cni/0.log" Jul 25 00:23:08.885927 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.885901 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/whereabouts-cni-bincopy/0.log" Jul 25 00:23:08.903434 ip-10-0-131-63 kubenswrapper[2586]: I0725 00:23:08.903403 2586 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vf4nl_dc8c23b0-c08b-4e85-a58a-e6f4d0bf7e6e/whereabouts-cni/0.log"