Jul 27 19:15:26.375617 ip-10-0-132-244 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 27 19:15:26.375632 ip-10-0-132-244 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 27 19:15:26.375644 ip-10-0-132-244 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 27 19:15:26.376011 ip-10-0-132-244 systemd[1]: Failed to start Kubernetes Kubelet. Jul 27 19:15:36.226345 ip-10-0-132-244 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 27 19:15:36.226360 ip-10-0-132-244 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 81d03d3d3ac74544b699194cbf24797c -- Jul 27 19:17:05.092457 ip-10-0-132-244 systemd[1]: Starting Kubernetes Kubelet... Jul 27 19:17:05.548735 ip-10-0-132-244 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 19:17:05.548735 ip-10-0-132-244 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 27 19:17:05.548735 ip-10-0-132-244 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 19:17:05.548735 ip-10-0-132-244 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 27 19:17:05.548735 ip-10-0-132-244 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 27 19:17:05.549629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.548815 2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 27 19:17:05.555561 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555526 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:17:05.555561 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555553 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:17:05.555561 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555559 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:17:05.555561 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555563 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:17:05.555561 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555567 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:17:05.555561 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555571 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555575 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555578 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555581 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555584 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555587 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555590 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555594 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555597 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555625 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555628 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555631 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555634 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555637 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555640 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555643 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555646 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555649 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555652 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555656 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:17:05.555817 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555660 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555662 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555665 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555669 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555672 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555674 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555677 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555679 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555682 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555685 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555687 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555690 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555692 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555695 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555698 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555701 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555704 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555706 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555709 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555712 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:17:05.556316 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555714 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555717 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555719 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555722 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555725 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555727 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555730 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555733 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555735 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555738 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555741 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555743 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555746 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555748 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555751 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555754 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555757 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555760 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555762 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555765 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:17:05.556927 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555768 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555770 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555774 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555776 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555779 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555781 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555784 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555786 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555789 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555792 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555794 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555797 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555800 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555802 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555805 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555807 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555810 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555813 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555816 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:17:05.557435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555819 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555822 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555825 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555828 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555830 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555833 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555836 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555838 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555841 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555843 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555845 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555848 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555851 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555854 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.555856 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556600 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556607 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556610 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556613 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556615 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:17:05.557901 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556618 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556621 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556624 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556627 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556629 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556632 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556634 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556637 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556640 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556642 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556645 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556648 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556651 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556654 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556666 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556669 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556672 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556675 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556678 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556681 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:17:05.558407 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556683 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556686 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556689 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556691 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556694 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556697 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556699 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556703 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556706 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556708 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556711 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556714 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556717 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556719 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556722 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556725 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556727 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556730 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556732 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556735 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:17:05.558906 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556738 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556740 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556743 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556745 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556747 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556751 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556754 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556756 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556761 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556765 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556768 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556772 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556774 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556777 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556780 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556785 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556789 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556792 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556795 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:17:05.559424 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556799 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556802 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556805 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556808 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556811 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556813 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556816 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556819 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556822 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556825 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556828 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556830 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556833 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556836 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556838 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556841 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556845 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556848 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556850 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:17:05.559913 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556853 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556856 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556859 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556862 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556864 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556867 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556870 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556872 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556875 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556878 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556880 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556883 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556885 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556888 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556891 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.556893 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557319 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557325 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557328 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557330 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:17:05.560415 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557333 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557336 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557339 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557341 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557344 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557346 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557349 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557352 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557355 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557357 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557360 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557363 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557366 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557369 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557371 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557373 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557377 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557379 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557382 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:17:05.560924 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557384 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557388 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557390 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557392 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557395 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557398 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557401 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557404 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557406 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557409 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557411 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557414 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557416 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557420 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557423 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557426 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557428 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557431 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557433 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557436 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:17:05.561413 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557439 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557442 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557444 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557447 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557450 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557452 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557455 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557458 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557460 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557463 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557465 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557467 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557477 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557480 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557483 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557485 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557488 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557490 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557494 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557496 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:17:05.561914 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557499 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557501 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557504 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557507 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557510 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557512 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557515 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557517 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557519 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557522 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557524 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557527 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557529 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557532 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557534 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557537 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557539 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557542 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557546 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557550 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:17:05.562426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557553 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557555 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557560 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557562 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557565 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557571 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557574 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557576 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557579 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557582 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557584 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557587 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557590 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557594 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557597 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.557600 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557678 2566 flags.go:64] FLAG: --address="0.0.0.0" Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557685 2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557692 2566 flags.go:64] FLAG: --anonymous-auth="true" Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557697 2566 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557701 2566 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 27 19:17:05.562902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557705 2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557717 2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557722 2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557725 2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557728 2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557732 2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557736 2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557739 2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557742 2566 flags.go:64] FLAG: --cgroup-root="" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557745 2566 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557748 2566 flags.go:64] FLAG: --client-ca-file="" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557752 2566 flags.go:64] FLAG: --cloud-provider="external" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557755 2566 flags.go:64] FLAG: --cluster-dns="[]" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557760 2566 flags.go:64] FLAG: --cluster-domain="" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557763 2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557766 2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557771 2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557775 2566 flags.go:64] FLAG: --container-log-max-files="5" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557779 2566 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557782 2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557785 2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557788 2566 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557791 2566 flags.go:64] FLAG: --contention-profiling="false" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557794 2566 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 27 19:17:05.563444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557797 2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557800 2566 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557803 2566 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557808 2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557811 2566 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557814 2566 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557817 2566 flags.go:64] FLAG: --enable-load-reader="false" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557820 2566 flags.go:64] FLAG: --enable-server="true" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557823 2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557828 2566 flags.go:64] FLAG: --event-burst="100" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557831 2566 flags.go:64] FLAG: --event-qps="50" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557834 2566 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557838 2566 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557840 2566 flags.go:64] FLAG: --eviction-hard="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557844 2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557847 2566 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557850 2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557853 2566 flags.go:64] FLAG: --eviction-soft="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557856 2566 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557858 2566 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557863 2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557866 2566 flags.go:64] FLAG: --experimental-mounter-path="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557869 2566 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557872 2566 flags.go:64] FLAG: --fail-swap-on="true" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557876 2566 flags.go:64] FLAG: --feature-gates="" Jul 27 19:17:05.564022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557880 2566 flags.go:64] FLAG: --file-check-frequency="20s" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557883 2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557886 2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557889 2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557892 2566 flags.go:64] FLAG: --healthz-port="10248" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557895 2566 flags.go:64] FLAG: --help="false" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557898 2566 flags.go:64] FLAG: --hostname-override="ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557901 2566 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557904 2566 flags.go:64] FLAG: --http-check-frequency="20s" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557907 2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557911 2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557914 2566 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557917 2566 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557920 2566 flags.go:64] FLAG: --image-service-endpoint="" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557923 2566 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557926 2566 flags.go:64] FLAG: --kube-api-burst="100" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557929 2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557932 2566 flags.go:64] FLAG: --kube-api-qps="50" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557935 2566 flags.go:64] FLAG: --kube-reserved="" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557938 2566 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557942 2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557946 2566 flags.go:64] FLAG: --kubelet-cgroups="" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557949 2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557952 2566 flags.go:64] FLAG: --lock-file="" Jul 27 19:17:05.564728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557954 2566 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557957 2566 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557960 2566 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557966 2566 flags.go:64] FLAG: --log-json-split-stream="false" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557970 2566 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557972 2566 flags.go:64] FLAG: --log-text-split-stream="false" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557975 2566 flags.go:64] FLAG: --logging-format="text" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557980 2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557983 2566 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557986 2566 flags.go:64] FLAG: --manifest-url="" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557989 2566 flags.go:64] FLAG: --manifest-url-header="" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557993 2566 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.557996 2566 flags.go:64] FLAG: --max-open-files="1000000" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558001 2566 flags.go:64] FLAG: --max-pods="110" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558004 2566 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558007 2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558010 2566 flags.go:64] FLAG: --memory-manager-policy="None" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558013 2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558016 2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558019 2566 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558022 2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558030 2566 flags.go:64] FLAG: --node-status-max-images="50" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558033 2566 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558036 2566 flags.go:64] FLAG: --oom-score-adj="-999" Jul 27 19:17:05.565347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558040 2566 flags.go:64] FLAG: --pod-cidr="" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558042 2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558048 2566 flags.go:64] FLAG: --pod-manifest-path="" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558051 2566 flags.go:64] FLAG: --pod-max-pids="-1" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558054 2566 flags.go:64] FLAG: --pods-per-core="0" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558059 2566 flags.go:64] FLAG: --port="10250" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558062 2566 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558065 2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b5ff9f199966720e" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558068 2566 flags.go:64] FLAG: --qos-reserved="" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558071 2566 flags.go:64] FLAG: --read-only-port="10255" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558074 2566 flags.go:64] FLAG: --register-node="true" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558077 2566 flags.go:64] FLAG: --register-with-taints="" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558080 2566 flags.go:64] FLAG: --registry-burst="10" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558084 2566 flags.go:64] FLAG: --registry-qps="5" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558087 2566 flags.go:64] FLAG: --reserved-cpus="" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558105 2566 flags.go:64] FLAG: --reserved-memory="" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558109 2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558112 2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558115 2566 flags.go:64] FLAG: --rotate-certificates="false" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558119 2566 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558122 2566 flags.go:64] FLAG: --runonce="false" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558125 2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558128 2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558131 2566 flags.go:64] FLAG: --seccomp-default="false" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558134 2566 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 27 19:17:05.566292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558137 2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558140 2566 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558144 2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558147 2566 flags.go:64] FLAG: --storage-driver-password="root" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558150 2566 flags.go:64] FLAG: --storage-driver-secure="false" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558153 2566 flags.go:64] FLAG: --storage-driver-table="stats" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558156 2566 flags.go:64] FLAG: --storage-driver-user="root" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558159 2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558162 2566 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558165 2566 flags.go:64] FLAG: --system-cgroups="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558167 2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558173 2566 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558177 2566 flags.go:64] FLAG: --tls-cert-file="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558180 2566 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558184 2566 flags.go:64] FLAG: --tls-min-version="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558187 2566 flags.go:64] FLAG: --tls-private-key-file="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558190 2566 flags.go:64] FLAG: --topology-manager-policy="none" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558193 2566 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558196 2566 flags.go:64] FLAG: --topology-manager-scope="container" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558199 2566 flags.go:64] FLAG: --v="2" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558204 2566 flags.go:64] FLAG: --version="false" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558209 2566 flags.go:64] FLAG: --vmodule="" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558214 2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.558218 2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558339 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:17:05.567402 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558342 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558345 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558348 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558351 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558353 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558356 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558360 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558363 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558366 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558369 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558372 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558375 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558377 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558380 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558383 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558385 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558388 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558390 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558393 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:17:05.568509 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558398 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558401 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558404 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558408 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558411 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558413 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558416 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558419 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558422 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558424 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558429 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558432 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558434 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558437 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558440 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558442 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558445 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558448 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558450 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:17:05.569366 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558453 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558455 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558458 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558460 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558463 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558466 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558468 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558471 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558473 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558476 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558479 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558482 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558484 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558488 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558490 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558493 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558496 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558498 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558501 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558503 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:17:05.569975 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558506 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558508 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558511 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558515 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558519 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558522 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558525 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558527 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558530 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558532 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558535 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558537 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558540 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558542 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558545 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558547 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558550 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558552 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558555 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558558 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:17:05.570615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558560 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558563 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558566 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558568 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558571 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558574 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558577 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558581 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558585 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558588 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558590 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558593 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558596 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558599 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558602 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558605 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558608 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558611 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558614 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:17:05.571435 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.558616 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.559459 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.567941 2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.567969 2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.567990 2566 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.568006 2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568116 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568126 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568133 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568137 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568142 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568147 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568152 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568156 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:17:05.572141 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568160 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568164 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568168 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568172 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568176 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568181 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568185 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568192 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568198 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568202 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568206 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568210 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568215 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568220 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568224 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568228 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568232 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568237 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568241 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568246 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:17:05.572516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568251 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568258 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568264 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568268 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568272 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568277 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568281 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568285 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568289 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568294 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568298 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568303 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568308 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568312 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568316 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568320 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568325 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568330 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568334 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:17:05.573184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568339 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568346 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568352 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568358 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568363 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568367 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568372 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568376 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568381 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568386 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568391 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568396 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568400 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568405 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568411 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568417 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568422 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568426 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568430 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:17:05.573644 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568435 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568440 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568444 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568448 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568452 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568457 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568461 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568465 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568469 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568473 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568478 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568482 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568486 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568490 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568495 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568499 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568504 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568508 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568512 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568516 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:17:05.574173 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568520 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568524 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568528 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568532 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568537 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568541 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568546 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568553 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568559 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568564 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568569 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568574 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568580 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.568589 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 19:17:05.574678 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568808 2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568819 2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568825 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568829 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568834 2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568838 2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568843 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568848 2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568853 2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568857 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568861 2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568866 2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568870 2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568875 2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568879 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568884 2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568888 2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568892 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568897 2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568901 2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 27 19:17:05.575117 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568905 2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568909 2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568913 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568918 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568922 2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568926 2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568931 2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568935 2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568940 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568945 2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568949 2566 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568953 2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568957 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568962 2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568966 2566 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568969 2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568973 2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568977 2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568981 2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568986 2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 27 19:17:05.575658 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568990 2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.568994 2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569001 2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569007 2566 feature_gate.go:328] unrecognized feature gate: Example2 Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569012 2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569017 2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569021 2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569026 2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569031 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569035 2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569040 2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569044 2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569048 2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569053 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569057 2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569061 2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569065 2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569069 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569074 2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569078 2566 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 27 19:17:05.576586 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569083 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569088 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569107 2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569112 2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569116 2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569121 2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569125 2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569129 2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569133 2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569138 2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569144 2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569150 2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569155 2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569160 2566 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569164 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569168 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569173 2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569177 2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569181 2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 27 19:17:05.577108 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569186 2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569190 2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569194 2566 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569199 2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569203 2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569207 2566 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569211 2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569215 2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569220 2566 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569224 2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569228 2566 feature_gate.go:328] unrecognized feature gate: Example Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569234 2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569238 2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569243 2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569247 2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569252 2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569257 2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569261 2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569266 2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 27 19:17:05.577870 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:05.569272 2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.569281 2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.569477 2566 server.go:962] "Client rotation is on, will bootstrap in background" Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.572781 2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.574641 2566 server.go:1019] "Starting client certificate rotation" Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.574760 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.574801 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 27 19:17:05.578629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.577888 2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 19:17:05.581424 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.581399 2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 27 19:17:05.598164 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.598128 2566 log.go:25] "Validated CRI v1 runtime API" Jul 27 19:17:05.601242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.601217 2566 log.go:25] "Validated CRI v1 image API" Jul 27 19:17:05.602131 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.602109 2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 27 19:17:05.606548 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.606504 2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b766df91-f5ad-4a27-8a65-758ed82616eb:/dev/nvme0n1p4 f0594a23-b882-4636-b50e-58d3e93b8957:/dev/nvme0n1p3] Jul 27 19:17:05.606548 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.606537 2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 27 19:17:05.611178 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.611153 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 19:17:05.612241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.612120 2566 manager.go:216] Machine: {Timestamp:2026-07-27 19:17:05.610581863 +0000 UTC m=+0.387743022 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104635 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25e9a8fcf0d0b14960c9eed7961240 SystemUUID:ec25e9a8-fcf0-d0b1-4960-c9eed7961240 BootID:81d03d3d-3ac7-4544-b699-194cbf24797c Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7b:ef:da:6f:e1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7b:ef:da:6f:e1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:fa:a4:57:7e:7c:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 27 19:17:05.612241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.612233 2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 27 19:17:05.612351 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.612331 2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 27 19:17:05.615140 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.615109 2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 27 19:17:05.615289 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.615144 2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-244.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 27 19:17:05.615981 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.615968 2566 topology_manager.go:138] "Creating topology manager with none policy" Jul 27 19:17:05.616016 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.615982 2566 container_manager_linux.go:309] "Creating device plugin manager" Jul 27 19:17:05.616016 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.615993 2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 19:17:05.616016 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.616008 2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 27 19:17:05.617282 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.617265 2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 27 19:17:05.620211 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620195 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 27 19:17:05.620355 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620344 2566 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 27 19:17:05.620393 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620381 2566 kubelet.go:486] "Attempting to sync node with API server" Jul 27 19:17:05.620393 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620392 2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 27 19:17:05.620459 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620408 2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 27 19:17:05.620459 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620417 2566 kubelet.go:398] "Adding apiserver pod source" Jul 27 19:17:05.620459 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620426 2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 27 19:17:05.620872 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620858 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 19:17:05.620938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.620883 2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 27 19:17:05.623311 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.623283 2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 27 19:17:05.626718 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.626686 2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 27 19:17:05.626903 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.626729 2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 27 19:17:05.628969 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.628951 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 27 19:17:05.629011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.628977 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 27 19:17:05.629011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.628987 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 27 19:17:05.629011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.628995 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 27 19:17:05.629011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629005 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 27 19:17:05.629011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629010 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 27 19:17:05.629236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629019 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 27 19:17:05.629236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629025 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 27 19:17:05.629236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629032 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 27 19:17:05.629236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629038 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 27 19:17:05.629236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629061 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 27 19:17:05.629236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629067 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 27 19:17:05.629980 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629965 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 27 19:17:05.629980 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.629979 2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 27 19:17:05.633369 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.633343 2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-244.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 27 19:17:05.633369 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.633361 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 27 19:17:05.633528 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.633338 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 27 19:17:05.633575 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.633558 2566 server.go:1268] "Started kubelet" Jul 27 19:17:05.633662 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.633629 2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 27 19:17:05.634124 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.634050 2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 27 19:17:05.634249 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.634147 2566 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 27 19:17:05.634534 ip-10-0-132-244 systemd[1]: Started Kubernetes Kubelet. Jul 27 19:17:05.635530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.635518 2566 server.go:310] "Adding debug handlers to kubelet server" Jul 27 19:17:05.636314 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.636300 2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 27 19:17:05.640383 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.639194 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-244.ec2.internal.18c63b97d71484ec default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-244.ec2.internal,UID:ip-10-0-132-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-244.ec2.internal,},FirstTimestamp:2026-07-27 19:17:05.63352702 +0000 UTC m=+0.410688236,LastTimestamp:2026-07-27 19:17:05.63352702 +0000 UTC m=+0.410688236,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-244.ec2.internal,}" Jul 27 19:17:05.640806 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.640782 2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 27 19:17:05.646022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.645995 2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 27 19:17:05.646022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.646013 2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 27 19:17:05.646500 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.646467 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:05.648175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.648145 2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 27 19:17:05.648645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.648392 2566 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 27 19:17:05.648645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.648408 2566 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 27 19:17:05.648645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.648554 2566 reconstruct.go:97] "Volume reconstruction finished" Jul 27 19:17:05.648645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.648564 2566 reconciler.go:29] "Reconciler: start to sync state" Jul 27 19:17:05.649147 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649128 2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 27 19:17:05.649147 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649148 2566 factory.go:55] Registering systemd factory Jul 27 19:17:05.649279 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649157 2566 factory.go:223] Registration of the systemd container factory successfully Jul 27 19:17:05.649479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649455 2566 factory.go:153] Registering CRI-O factory Jul 27 19:17:05.649479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649477 2566 factory.go:223] Registration of the crio container factory successfully Jul 27 19:17:05.649624 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649506 2566 factory.go:103] Registering Raw factory Jul 27 19:17:05.649624 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.649528 2566 manager.go:1196] Started watching for new ooms in manager Jul 27 19:17:05.650049 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.650036 2566 manager.go:349] Starting recovery of all containers Jul 27 19:17:05.654586 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.654551 2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-244.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 27 19:17:05.654746 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.654615 2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 27 19:17:05.656730 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.656649 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 27 19:17:05.662012 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.661919 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6nb5d" Jul 27 19:17:05.664290 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.664268 2566 manager.go:354] Recovery completed Jul 27 19:17:05.668293 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.668267 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6nb5d" Jul 27 19:17:05.668803 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.668787 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.670807 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.670785 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.670913 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.670814 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.670913 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.670824 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.671402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.671384 2566 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 27 19:17:05.671402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.671401 2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 27 19:17:05.671530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.671414 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.671530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.671421 2566 state_mem.go:36] "Initialized new in-memory state store" Jul 27 19:17:05.672117 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.672085 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.672216 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.672130 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.672216 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.672144 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.672692 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.672595 2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-244.ec2.internal.18c63b97d94d504f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-244.ec2.internal,UID:ip-10-0-132-244.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-244.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-244.ec2.internal,},FirstTimestamp:2026-07-27 19:17:05.670803535 +0000 UTC m=+0.447964697,LastTimestamp:2026-07-27 19:17:05.670803535 +0000 UTC m=+0.447964697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-244.ec2.internal,}" Jul 27 19:17:05.675150 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.675137 2566 policy_none.go:49] "None policy: Start" Jul 27 19:17:05.675212 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.675156 2566 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 27 19:17:05.675212 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.675168 2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 27 19:17:05.677182 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.677169 2566 policy_none.go:47] "Start" Jul 27 19:17:05.716850 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.716831 2566 manager.go:337] "Starting Device Plugin manager" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.716867 2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.716878 2566 server.go:87] "Starting device plugin registration server" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.717203 2566 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.717215 2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.717300 2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.717392 2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.717400 2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.717994 2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 27 19:17:05.735166 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.718275 2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:05.756292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.756268 2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 27 19:17:05.756388 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.756314 2566 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 27 19:17:05.756388 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.756339 2566 kubelet.go:2448] "Starting kubelet main sync loop" Jul 27 19:17:05.756388 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.756378 2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 27 19:17:05.760778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.760755 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:17:05.818049 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.817974 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.821592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.821573 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.821705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.821605 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.821705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.821616 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.821705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.821648 2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.831154 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.831128 2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.831234 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.831158 2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-244.ec2.internal\": node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:05.853241 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.853209 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:05.857356 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.857329 2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal"] Jul 27 19:17:05.857486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.857413 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.858466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.858449 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.858579 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.858477 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.858579 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.858486 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.861063 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861044 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.861204 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861188 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.861267 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861225 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.861915 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861898 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.861915 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861907 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.862036 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861931 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.862036 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861943 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.862036 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861960 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.862036 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.861945 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.864370 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.864350 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.864493 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.864383 2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 27 19:17:05.866432 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.866409 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientMemory" Jul 27 19:17:05.866539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.866451 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasNoDiskPressure" Jul 27 19:17:05.866539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.866470 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeHasSufficientPID" Jul 27 19:17:05.889042 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.889002 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-244.ec2.internal\" not found" node="ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.893682 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.893657 2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-244.ec2.internal\" not found" node="ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.950984 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.950942 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d9fa04e8305f63fcc41b14bd23fdc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"22d9fa04e8305f63fcc41b14bd23fdc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.950984 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.950979 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dbb63acc39cffb9179e67a48cdbfda3-config\") pod \"kube-apiserver-proxy-ip-10-0-132-244.ec2.internal\" (UID: \"4dbb63acc39cffb9179e67a48cdbfda3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.951213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:05.950998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d9fa04e8305f63fcc41b14bd23fdc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"22d9fa04e8305f63fcc41b14bd23fdc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:05.954207 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:05.954189 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:06.052090 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.052039 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d9fa04e8305f63fcc41b14bd23fdc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"22d9fa04e8305f63fcc41b14bd23fdc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.052090 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.052090 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d9fa04e8305f63fcc41b14bd23fdc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"22d9fa04e8305f63fcc41b14bd23fdc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.052241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.052123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dbb63acc39cffb9179e67a48cdbfda3-config\") pod \"kube-apiserver-proxy-ip-10-0-132-244.ec2.internal\" (UID: \"4dbb63acc39cffb9179e67a48cdbfda3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.052241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.052155 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4dbb63acc39cffb9179e67a48cdbfda3-config\") pod \"kube-apiserver-proxy-ip-10-0-132-244.ec2.internal\" (UID: \"4dbb63acc39cffb9179e67a48cdbfda3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.052241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.052154 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/22d9fa04e8305f63fcc41b14bd23fdc0-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"22d9fa04e8305f63fcc41b14bd23fdc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.052241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.052154 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/22d9fa04e8305f63fcc41b14bd23fdc0-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal\" (UID: \"22d9fa04e8305f63fcc41b14bd23fdc0\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.055171 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.055148 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:06.156177 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.156142 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:06.191294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.191268 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.197579 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.197561 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.256532 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.256468 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:06.356634 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.356584 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:06.457366 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.457278 2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-244.ec2.internal\" not found" Jul 27 19:17:06.466829 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.466805 2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:17:06.546698 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.546662 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.562042 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.562018 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 19:17:06.563528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.563506 2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" Jul 27 19:17:06.572640 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.572609 2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 27 19:17:06.573679 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.573654 2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 27 19:17:06.573814 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.573797 2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 27 19:17:06.620753 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.620716 2566 apiserver.go:52] "Watching apiserver" Jul 27 19:17:06.629072 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.629042 2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 27 19:17:06.629437 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.629413 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-vnw5n","openshift-cluster-node-tuning-operator/tuned-w7q6w","openshift-dns/node-resolver-4wrcm","openshift-image-registry/node-ca-hnnwt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal","openshift-multus/multus-8ff5w","openshift-multus/multus-additional-cni-plugins-j54z8","openshift-multus/network-metrics-daemon-tqrkg","kube-system/konnectivity-agent-qn58b","kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv","openshift-network-diagnostics/network-check-target-ccflr","openshift-network-operator/iptables-alerter-6sgs4","openshift-ovn-kubernetes/ovnkube-node-7t8np"] Jul 27 19:17:06.631741 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.631722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:06.631820 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.631801 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:06.633884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.633866 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.635870 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.635845 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.635982 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.635897 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k4z9d\"" Jul 27 19:17:06.635982 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.635856 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.636172 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.636153 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.638288 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.638267 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.638288 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.638280 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.638456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.638348 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-dxmng\"" Jul 27 19:17:06.638456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.638366 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.638456 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.638434 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:06.640739 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.640719 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.642812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.642791 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 27 19:17:06.642917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.642798 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 27 19:17:06.642917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.642825 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 27 19:17:06.642917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.642850 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 27 19:17:06.643066 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.643054 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.643156 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.643142 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.643384 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.643368 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-bg7k7\"" Jul 27 19:17:06.643430 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.643420 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.645569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.645549 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 27 19:17:06.645644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.645581 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 27 19:17:06.645644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.645605 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.645723 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.645661 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nzmm5\"" Jul 27 19:17:06.645723 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.645683 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.646163 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.646147 2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 27 19:17:06.646230 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.646174 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.648207 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.648187 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 27 19:17:06.648319 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.648252 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 27 19:17:06.648375 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.648323 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bfdth\"" Jul 27 19:17:06.648820 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.648804 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:06.648929 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.648889 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:06.651450 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.651428 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.654160 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.654133 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.654789 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.654773 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 27 19:17:06.654876 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.654811 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 27 19:17:06.654876 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.654823 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wjl2k\"" Jul 27 19:17:06.655087 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655070 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-k8s-cni-cncf-io\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.655169 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655114 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-daemon-config\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.655169 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655146 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-systemd-units\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655179 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovn-node-metrics-cert\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdr2p\" (UniqueName: \"kubernetes.io/projected/d4fa595d-8a37-41fd-b022-91a1a0d94b85-kube-api-access-mdr2p\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.655260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655224 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-log-socket\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-run-ovn-kubernetes\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655261 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysctl-conf\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655276 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4fa595d-8a37-41fd-b022-91a1a0d94b85-tmp\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655291 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-ovn\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655312 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfae84b-f178-4938-a7db-12bc33b72023-kubelet-config\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655326 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-system-cni-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655341 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-kubelet\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655355 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-systemd\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655394 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-cni-netd\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655410 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655439 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-etc-kubernetes\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq7sc\" (UniqueName: \"kubernetes.io/projected/dfedf762-12a2-4461-999d-071931cd94c7-kube-api-access-qq7sc\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655482 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-tuned\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655523 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-slash\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655553 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-cni-bin\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655580 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-cnibin\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655603 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-hosts-file\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655639 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.655688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655677 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-cni-binary-copy\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655748 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655775 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysctl-d\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655799 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovnkube-script-lib\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655844 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655872 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-netns\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655897 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-os-release\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655922 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-os-release\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655947 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-cni-multus\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.655984 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9hth\" (UniqueName: \"kubernetes.io/projected/4f1f1c0f-5832-423d-9bb5-7701a6e87553-kube-api-access-m9hth\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656045 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-run-netns\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656073 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-system-cni-dir\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.656122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656115 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656141 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-cni-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhv2m\" (UniqueName: \"kubernetes.io/projected/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-kube-api-access-zhv2m\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-lib-modules\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656220 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovnkube-config\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656242 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-host\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-env-overrides\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-conf-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-multus-certs\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656372 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-modprobe-d\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656406 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57r8\" (UniqueName: \"kubernetes.io/projected/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-kube-api-access-l57r8\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656433 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfae84b-f178-4938-a7db-12bc33b72023-dbus\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656478 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-cni-bin\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656498 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-hostroot\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656522 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656550 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.656651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656545 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-kubelet\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.656856 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657034 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-etc-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657130 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-var-lib-kubelet\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657190 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-kubernetes\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657237 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-systemd\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657261 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657273 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-run\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.657366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657333 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-tmp-dir\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.657770 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657555 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-node-log\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.657770 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657600 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-cnibin\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.657770 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657679 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysconfig\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.657770 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657720 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-sys\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.657770 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657725 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.657993 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.657927 2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 27 19:17:06.658071 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.658048 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-q9rgl\"" Jul 27 19:17:06.658599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.658578 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-var-lib-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.658709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.658619 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjmz9\" (UniqueName: \"kubernetes.io/projected/6136915b-a655-4dc2-9d87-51ada7dc77a6-kube-api-access-pjmz9\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.658709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.658660 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-socket-dir-parent\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.660291 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.660088 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.660708 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.660645 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 27 19:17:06.661379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.661355 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.661485 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.661439 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnbx4\"" Jul 27 19:17:06.661485 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.661458 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.663665 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.663646 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 27 19:17:06.663769 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.663648 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-7wz2l\"" Jul 27 19:17:06.663769 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.663652 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 27 19:17:06.663918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.663902 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 27 19:17:06.675241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.675197 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-26 19:12:07 +0000 UTC" deadline="2028-04-07 11:06:15.473746992 +0000 UTC" Jul 27 19:17:06.675241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.675236 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14871h49m8.798512934s" Jul 27 19:17:06.736915 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.736892 2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mv5hb" Jul 27 19:17:06.747148 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.747114 2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mv5hb" Jul 27 19:17:06.749005 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.748981 2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 27 19:17:06.758932 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.758900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovn-node-metrics-cert\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.758938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dhkl\" (UniqueName: \"kubernetes.io/projected/18f93f18-ce46-4be6-8870-9bb9db422b11-kube-api-access-9dhkl\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.758958 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-sys-fs\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.758974 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdr2p\" (UniqueName: \"kubernetes.io/projected/d4fa595d-8a37-41fd-b022-91a1a0d94b85-kube-api-access-mdr2p\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.758991 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-log-socket\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-run-ovn-kubernetes\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-log-socket\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759080 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sx6n\" (UniqueName: \"kubernetes.io/projected/bc30b7bb-1847-4164-91b6-338f667bdf24-kube-api-access-2sx6n\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759114 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-run-ovn-kubernetes\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759130 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysctl-conf\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4fa595d-8a37-41fd-b022-91a1a0d94b85-tmp\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759218 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-ovn\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759261 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfae84b-f178-4938-a7db-12bc33b72023-kubelet-config\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759294 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-device-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759307 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-ovn\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759321 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-system-cni-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759359 2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759385 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfae84b-f178-4938-a7db-12bc33b72023-kubelet-config\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysctl-conf\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759391 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-system-cni-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-kubelet\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759432 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-kubelet\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759467 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-systemd\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759504 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-cni-netd\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759513 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-systemd\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759547 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759553 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-cni-netd\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.759626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759585 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759628 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-etc-kubernetes\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759656 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq7sc\" (UniqueName: \"kubernetes.io/projected/dfedf762-12a2-4461-999d-071931cd94c7-kube-api-access-qq7sc\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759682 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-tuned\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759701 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-etc-kubernetes\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-slash\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759748 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-slash\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-cni-bin\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.759798 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759807 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-cnibin\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.759885 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:07.259863237 +0000 UTC m=+2.037024402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759927 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-cni-bin\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759958 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-hosts-file\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759988 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.759997 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-cnibin\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760013 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-cni-binary-copy\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760037 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:06.760444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760056 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-hosts-file\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysctl-d\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760085 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760123 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovnkube-script-lib\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760153 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760178 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-netns\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760204 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760227 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-os-release\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760251 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-os-release\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760287 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-cni-multus\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760335 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48a0d645-1915-42a5-8aea-648f94ea0190-konnectivity-ca\") pod \"konnectivity-agent-qn58b\" (UID: \"48a0d645-1915-42a5-8aea-648f94ea0190\") " pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760297 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-netns\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760408 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9hth\" (UniqueName: \"kubernetes.io/projected/4f1f1c0f-5832-423d-9bb5-7701a6e87553-kube-api-access-m9hth\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760438 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-registration-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760443 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysctl-d\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760464 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-etc-selinux\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.761251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760508 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-os-release\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760565 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-run-netns\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760596 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-cni-binary-copy\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760628 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-run-netns\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760647 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-run-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760686 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-os-release\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-system-cni-dir\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760719 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760744 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-cni-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.760757 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760631 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-cni-multus\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760792 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-system-cni-dir\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.760823 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:07.260806766 +0000 UTC m=+2.037967917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760829 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovnkube-script-lib\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhv2m\" (UniqueName: \"kubernetes.io/projected/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-kube-api-access-zhv2m\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760869 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/663cc638-8629-41a2-81f3-e8bc031b6bb9-iptables-alerter-script\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.762028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760980 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-cni-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.760990 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trlf\" (UniqueName: \"kubernetes.io/projected/663cc638-8629-41a2-81f3-e8bc031b6bb9-kube-api-access-9trlf\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761121 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-lib-modules\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovnkube-config\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761194 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/663cc638-8629-41a2-81f3-e8bc031b6bb9-host-slash\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761234 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-host\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761244 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-lib-modules\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761396 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-env-overrides\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-conf-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761438 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6136915b-a655-4dc2-9d87-51ada7dc77a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761708 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovnkube-config\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761833 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-conf-dir\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761851 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/4f1f1c0f-5832-423d-9bb5-7701a6e87553-env-overrides\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-multus-certs\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761907 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-host\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761950 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-socket-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.761996 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-multus-certs\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.762795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762047 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-modprobe-d\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57r8\" (UniqueName: \"kubernetes.io/projected/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-kube-api-access-l57r8\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762151 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-modprobe-d\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762241 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762290 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762290 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfae84b-f178-4938-a7db-12bc33b72023-dbus\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfae84b-f178-4938-a7db-12bc33b72023-dbus\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-cni-bin\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762428 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-hostroot\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762496 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-cni-bin\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762496 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762534 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-kubelet\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762541 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-hostroot\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762574 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18f93f18-ce46-4be6-8870-9bb9db422b11-host\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762606 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48a0d645-1915-42a5-8aea-648f94ea0190-agent-certs\") pod \"konnectivity-agent-qn58b\" (UID: \"48a0d645-1915-42a5-8aea-648f94ea0190\") " pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762671 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6136915b-a655-4dc2-9d87-51ada7dc77a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762678 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-var-lib-kubelet\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.763329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762786 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-etc-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762812 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-var-lib-kubelet\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762835 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-kubernetes\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762855 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-systemd\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-run\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762910 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-tmp-dir\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762935 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-node-log\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762967 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-cnibin\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.762990 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysconfig\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763009 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-sys\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763031 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-var-lib-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763055 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjmz9\" (UniqueName: \"kubernetes.io/projected/6136915b-a655-4dc2-9d87-51ada7dc77a6-kube-api-access-pjmz9\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763078 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-socket-dir-parent\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763118 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-k8s-cni-cncf-io\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763138 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-run\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-daemon-config\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18f93f18-ce46-4be6-8870-9bb9db422b11-serviceca\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763251 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-etc-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763296 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-kubernetes\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763334 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-systemd\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763363 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-systemd-units\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-systemd-units\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763462 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-node-log\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763561 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-var-lib-kubelet\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763628 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-socket-dir-parent\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763669 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-cnibin\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763687 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-multus-daemon-config\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-sysconfig\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763746 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-host-run-k8s-cni-cncf-io\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763759 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-tmp-dir\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763765 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d4fa595d-8a37-41fd-b022-91a1a0d94b85-etc-tuned\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763785 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d4fa595d-8a37-41fd-b022-91a1a0d94b85-sys\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763828 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/4f1f1c0f-5832-423d-9bb5-7701a6e87553-var-lib-openvswitch\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.763902 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/4f1f1c0f-5832-423d-9bb5-7701a6e87553-ovn-node-metrics-cert\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.764555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.764154 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d4fa595d-8a37-41fd-b022-91a1a0d94b85-tmp\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.772191 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.772160 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:06.772314 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.772193 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:06.772314 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.772221 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:06.772314 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:06.772277 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:07.272262643 +0000 UTC m=+2.049423789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:06.774441 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.774404 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdr2p\" (UniqueName: \"kubernetes.io/projected/d4fa595d-8a37-41fd-b022-91a1a0d94b85-kube-api-access-mdr2p\") pod \"tuned-w7q6w\" (UID: \"d4fa595d-8a37-41fd-b022-91a1a0d94b85\") " pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.774710 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.774687 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq7sc\" (UniqueName: \"kubernetes.io/projected/dfedf762-12a2-4461-999d-071931cd94c7-kube-api-access-qq7sc\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:06.776577 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.776549 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57r8\" (UniqueName: \"kubernetes.io/projected/57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4-kube-api-access-l57r8\") pod \"node-resolver-4wrcm\" (UID: \"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4\") " pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:06.776790 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.776772 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9hth\" (UniqueName: \"kubernetes.io/projected/4f1f1c0f-5832-423d-9bb5-7701a6e87553-kube-api-access-m9hth\") pod \"ovnkube-node-7t8np\" (UID: \"4f1f1c0f-5832-423d-9bb5-7701a6e87553\") " pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:06.777151 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.777130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhv2m\" (UniqueName: \"kubernetes.io/projected/a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0-kube-api-access-zhv2m\") pod \"multus-8ff5w\" (UID: \"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0\") " pod="openshift-multus/multus-8ff5w" Jul 27 19:17:06.780544 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.780520 2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:17:06.781297 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.781277 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjmz9\" (UniqueName: \"kubernetes.io/projected/6136915b-a655-4dc2-9d87-51ada7dc77a6-kube-api-access-pjmz9\") pod \"multus-additional-cni-plugins-j54z8\" (UID: \"6136915b-a655-4dc2-9d87-51ada7dc77a6\") " pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:06.817548 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:06.817506 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbb63acc39cffb9179e67a48cdbfda3.slice/crio-0d9881241bfd212c9f2ef711889123e7da4cbe893b04d8c68396dab40d8f893b WatchSource:0}: Error finding container 0d9881241bfd212c9f2ef711889123e7da4cbe893b04d8c68396dab40d8f893b: Status 404 returned error can't find the container with id 0d9881241bfd212c9f2ef711889123e7da4cbe893b04d8c68396dab40d8f893b Jul 27 19:17:06.819391 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:06.819355 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22d9fa04e8305f63fcc41b14bd23fdc0.slice/crio-491c58e03aa5e2ae3e6e1430967b65ffbb635421329a538c0e6142fdc2c9e988 WatchSource:0}: Error finding container 491c58e03aa5e2ae3e6e1430967b65ffbb635421329a538c0e6142fdc2c9e988: Status 404 returned error can't find the container with id 491c58e03aa5e2ae3e6e1430967b65ffbb635421329a538c0e6142fdc2c9e988 Jul 27 19:17:06.823704 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.823683 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:17:06.864041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864001 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/663cc638-8629-41a2-81f3-e8bc031b6bb9-host-slash\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.864041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-socket-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864130 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/663cc638-8629-41a2-81f3-e8bc031b6bb9-host-slash\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18f93f18-ce46-4be6-8870-9bb9db422b11-host\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864208 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48a0d645-1915-42a5-8aea-648f94ea0190-agent-certs\") pod \"konnectivity-agent-qn58b\" (UID: \"48a0d645-1915-42a5-8aea-648f94ea0190\") " pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-socket-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864239 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18f93f18-ce46-4be6-8870-9bb9db422b11-serviceca\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dhkl\" (UniqueName: \"kubernetes.io/projected/18f93f18-ce46-4be6-8870-9bb9db422b11-kube-api-access-9dhkl\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.864269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864266 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/18f93f18-ce46-4be6-8870-9bb9db422b11-host\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864286 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-sys-fs\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864313 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sx6n\" (UniqueName: \"kubernetes.io/projected/bc30b7bb-1847-4164-91b6-338f667bdf24-kube-api-access-2sx6n\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864342 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-device-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864358 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-sys-fs\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48a0d645-1915-42a5-8aea-648f94ea0190-konnectivity-ca\") pod \"konnectivity-agent-qn58b\" (UID: \"48a0d645-1915-42a5-8aea-648f94ea0190\") " pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-registration-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864490 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-etc-selinux\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864512 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-kubelet-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864518 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/663cc638-8629-41a2-81f3-e8bc031b6bb9-iptables-alerter-script\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trlf\" (UniqueName: \"kubernetes.io/projected/663cc638-8629-41a2-81f3-e8bc031b6bb9-kube-api-access-9trlf\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.864602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864593 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-registration-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.865215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864633 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-device-dir\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.865215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.864648 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/bc30b7bb-1847-4164-91b6-338f667bdf24-etc-selinux\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.865215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.865049 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48a0d645-1915-42a5-8aea-648f94ea0190-konnectivity-ca\") pod \"konnectivity-agent-qn58b\" (UID: \"48a0d645-1915-42a5-8aea-648f94ea0190\") " pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.865215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.865066 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/663cc638-8629-41a2-81f3-e8bc031b6bb9-iptables-alerter-script\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.865440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.865423 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/18f93f18-ce46-4be6-8870-9bb9db422b11-serviceca\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.866595 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.866580 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48a0d645-1915-42a5-8aea-648f94ea0190-agent-certs\") pod \"konnectivity-agent-qn58b\" (UID: \"48a0d645-1915-42a5-8aea-648f94ea0190\") " pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:06.871493 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.871471 2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:17:06.880248 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.880216 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dhkl\" (UniqueName: \"kubernetes.io/projected/18f93f18-ce46-4be6-8870-9bb9db422b11-kube-api-access-9dhkl\") pod \"node-ca-hnnwt\" (UID: \"18f93f18-ce46-4be6-8870-9bb9db422b11\") " pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:06.880371 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.880215 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trlf\" (UniqueName: \"kubernetes.io/projected/663cc638-8629-41a2-81f3-e8bc031b6bb9-kube-api-access-9trlf\") pod \"iptables-alerter-6sgs4\" (UID: \"663cc638-8629-41a2-81f3-e8bc031b6bb9\") " pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:06.881343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.881326 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sx6n\" (UniqueName: \"kubernetes.io/projected/bc30b7bb-1847-4164-91b6-338f667bdf24-kube-api-access-2sx6n\") pod \"aws-ebs-csi-driver-node-977mv\" (UID: \"bc30b7bb-1847-4164-91b6-338f667bdf24\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:06.976511 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.976422 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" Jul 27 19:17:06.986213 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:06.986182 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fa595d_8a37_41fd_b022_91a1a0d94b85.slice/crio-9cc7c4475a49686b9e05680f60bd81dce215179768831d5a736f6baec8c6aa88 WatchSource:0}: Error finding container 9cc7c4475a49686b9e05680f60bd81dce215179768831d5a736f6baec8c6aa88: Status 404 returned error can't find the container with id 9cc7c4475a49686b9e05680f60bd81dce215179768831d5a736f6baec8c6aa88 Jul 27 19:17:06.994470 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:06.994445 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4wrcm" Jul 27 19:17:07.003778 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.003749 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f91ac2_c9b3_4a51_a772_65ec3dc4e0c4.slice/crio-ee1a2ebcda4bd4d5da2f9167d8d6909abc6fe34fbf27a52e36a7c2accfe31ee2 WatchSource:0}: Error finding container ee1a2ebcda4bd4d5da2f9167d8d6909abc6fe34fbf27a52e36a7c2accfe31ee2: Status 404 returned error can't find the container with id ee1a2ebcda4bd4d5da2f9167d8d6909abc6fe34fbf27a52e36a7c2accfe31ee2 Jul 27 19:17:07.008207 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.008185 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:07.013053 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.013032 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8ff5w" Jul 27 19:17:07.016557 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.016525 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1f1c0f_5832_423d_9bb5_7701a6e87553.slice/crio-b7474d18ffffa4a510b4e7c8c5e1617c0093d6ab818c376f2ad19b408a31f6b9 WatchSource:0}: Error finding container b7474d18ffffa4a510b4e7c8c5e1617c0093d6ab818c376f2ad19b408a31f6b9: Status 404 returned error can't find the container with id b7474d18ffffa4a510b4e7c8c5e1617c0093d6ab818c376f2ad19b408a31f6b9 Jul 27 19:17:07.021364 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.021334 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6cd3b48_be1c_4d7c_aadb_4f6839cd75b0.slice/crio-057d06dc64928c5a1bb9a42a10f5fef5d630efc20ce4551d587af4a0d41b7a68 WatchSource:0}: Error finding container 057d06dc64928c5a1bb9a42a10f5fef5d630efc20ce4551d587af4a0d41b7a68: Status 404 returned error can't find the container with id 057d06dc64928c5a1bb9a42a10f5fef5d630efc20ce4551d587af4a0d41b7a68 Jul 27 19:17:07.028287 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.028263 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-j54z8" Jul 27 19:17:07.033187 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.033159 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:07.038244 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.038212 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6136915b_a655_4dc2_9d87_51ada7dc77a6.slice/crio-fe66faa580f5435b42b76cf73c6297dbedae97a5c87659ed78fd79bb6221bff9 WatchSource:0}: Error finding container fe66faa580f5435b42b76cf73c6297dbedae97a5c87659ed78fd79bb6221bff9: Status 404 returned error can't find the container with id fe66faa580f5435b42b76cf73c6297dbedae97a5c87659ed78fd79bb6221bff9 Jul 27 19:17:07.042005 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.041980 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a0d645_1915_42a5_8aea_648f94ea0190.slice/crio-18198a19d6a72cafac5ad438ed1067ed018cfa2dc5561529480856c311108b19 WatchSource:0}: Error finding container 18198a19d6a72cafac5ad438ed1067ed018cfa2dc5561529480856c311108b19: Status 404 returned error can't find the container with id 18198a19d6a72cafac5ad438ed1067ed018cfa2dc5561529480856c311108b19 Jul 27 19:17:07.060488 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.060454 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6sgs4" Jul 27 19:17:07.070361 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.070331 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663cc638_8629_41a2_81f3_e8bc031b6bb9.slice/crio-34827f2e55627de55ce02ffd69b02d6fd35f0a200f87ccf35d21c0f6fe6b9752 WatchSource:0}: Error finding container 34827f2e55627de55ce02ffd69b02d6fd35f0a200f87ccf35d21c0f6fe6b9752: Status 404 returned error can't find the container with id 34827f2e55627de55ce02ffd69b02d6fd35f0a200f87ccf35d21c0f6fe6b9752 Jul 27 19:17:07.073604 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.073579 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" Jul 27 19:17:07.082051 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.082024 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc30b7bb_1847_4164_91b6_338f667bdf24.slice/crio-267d25ae478112aadcc1163a462f77de7285c2cb945fd2108f9b871e300d0789 WatchSource:0}: Error finding container 267d25ae478112aadcc1163a462f77de7285c2cb945fd2108f9b871e300d0789: Status 404 returned error can't find the container with id 267d25ae478112aadcc1163a462f77de7285c2cb945fd2108f9b871e300d0789 Jul 27 19:17:07.090038 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.090011 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hnnwt" Jul 27 19:17:07.099931 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:07.099900 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18f93f18_ce46_4be6_8870_9bb9db422b11.slice/crio-e20099e242030dd5ea94fe47c0bc4f10846406bbd97ca90bb8b7c2db3a5c7cea WatchSource:0}: Error finding container e20099e242030dd5ea94fe47c0bc4f10846406bbd97ca90bb8b7c2db3a5c7cea: Status 404 returned error can't find the container with id e20099e242030dd5ea94fe47c0bc4f10846406bbd97ca90bb8b7c2db3a5c7cea Jul 27 19:17:07.268528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.268430 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:07.268528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.268487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:07.268742 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.268563 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:07.268742 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.268606 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:07.268742 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.268647 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:08.268622222 +0000 UTC m=+3.045783370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:07.268742 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.268678 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:08.268667019 +0000 UTC m=+3.045828171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:07.369219 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.369170 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:07.369403 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.369344 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:07.369403 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.369368 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:07.369403 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.369381 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:07.369536 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:07.369443 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:08.369423867 +0000 UTC m=+3.146585014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:07.620635 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.620151 2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 27 19:17:07.749926 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.749848 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:12:08 +0000 UTC" deadline="2028-04-26 13:07:37.419066924 +0000 UTC" Jul 27 19:17:07.749926 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.749889 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15329h50m29.669182873s" Jul 27 19:17:07.789038 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.788875 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hnnwt" event={"ID":"18f93f18-ce46-4be6-8870-9bb9db422b11","Type":"ContainerStarted","Data":"e20099e242030dd5ea94fe47c0bc4f10846406bbd97ca90bb8b7c2db3a5c7cea"} Jul 27 19:17:07.791028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.790915 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" event={"ID":"bc30b7bb-1847-4164-91b6-338f667bdf24","Type":"ContainerStarted","Data":"267d25ae478112aadcc1163a462f77de7285c2cb945fd2108f9b871e300d0789"} Jul 27 19:17:07.807907 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.807799 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wrcm" event={"ID":"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4","Type":"ContainerStarted","Data":"ee1a2ebcda4bd4d5da2f9167d8d6909abc6fe34fbf27a52e36a7c2accfe31ee2"} Jul 27 19:17:07.815203 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.815136 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" event={"ID":"d4fa595d-8a37-41fd-b022-91a1a0d94b85","Type":"ContainerStarted","Data":"9cc7c4475a49686b9e05680f60bd81dce215179768831d5a736f6baec8c6aa88"} Jul 27 19:17:07.829478 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.829433 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" event={"ID":"4dbb63acc39cffb9179e67a48cdbfda3","Type":"ContainerStarted","Data":"0d9881241bfd212c9f2ef711889123e7da4cbe893b04d8c68396dab40d8f893b"} Jul 27 19:17:07.851422 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.848838 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"22d9fa04e8305f63fcc41b14bd23fdc0","Type":"ContainerStarted","Data":"491c58e03aa5e2ae3e6e1430967b65ffbb635421329a538c0e6142fdc2c9e988"} Jul 27 19:17:07.866477 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.866434 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6sgs4" event={"ID":"663cc638-8629-41a2-81f3-e8bc031b6bb9","Type":"ContainerStarted","Data":"34827f2e55627de55ce02ffd69b02d6fd35f0a200f87ccf35d21c0f6fe6b9752"} Jul 27 19:17:07.885167 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:07.885053 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qn58b" event={"ID":"48a0d645-1915-42a5-8aea-648f94ea0190","Type":"ContainerStarted","Data":"18198a19d6a72cafac5ad438ed1067ed018cfa2dc5561529480856c311108b19"} Jul 27 19:17:09.686906 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:09.686863 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerStarted","Data":"fe66faa580f5435b42b76cf73c6297dbedae97a5c87659ed78fd79bb6221bff9"} Jul 27 19:17:09.701990 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:09.701949 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8ff5w" event={"ID":"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0","Type":"ContainerStarted","Data":"057d06dc64928c5a1bb9a42a10f5fef5d630efc20ce4551d587af4a0d41b7a68"} Jul 27 19:17:09.707180 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:09.707122 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"b7474d18ffffa4a510b4e7c8c5e1617c0093d6ab818c376f2ad19b408a31f6b9"} Jul 27 19:17:10.059589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.059545 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:10.059843 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.059604 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:10.059843 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.059745 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:10.059843 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.059822 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:12.059802019 +0000 UTC m=+5.054298977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:10.060260 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.060243 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:10.060319 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.060294 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:12.060279473 +0000 UTC m=+5.054776435 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:10.160387 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.160326 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:10.160576 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.160539 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:10.160576 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.160558 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:10.160576 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.160571 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:10.160738 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.160628 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:12.160610424 +0000 UTC m=+5.155107391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:10.533467 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.533120 2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-26 19:12:08 +0000 UTC" deadline="2028-01-30 22:32:51.842321204 +0000 UTC" Jul 27 19:17:10.533467 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.533183 2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13251h15m41.309142099s" Jul 27 19:17:10.540471 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.539603 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:10.540471 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.539748 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:10.540471 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.540190 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:10.540471 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.540290 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:10.540471 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:10.540353 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:10.540471 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:10.540421 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:12.076361 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:12.076275 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:12.076361 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:12.076335 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:12.077029 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.076476 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:12.077029 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.076543 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:16.076521205 +0000 UTC m=+9.071018176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:12.077029 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.076946 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:12.077029 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.076995 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:16.076981247 +0000 UTC m=+9.071478205 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:12.176730 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:12.176691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:12.177440 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.176954 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:12.177440 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.176978 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:12.177440 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.176993 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:12.177440 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.177052 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:16.177033931 +0000 UTC m=+9.171530889 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:12.539588 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:12.539513 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:12.539863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:12.539663 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:12.539863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.539675 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:12.539863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.539791 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:12.539863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:12.539848 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:12.540085 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:12.539948 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:14.539540 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:14.539502 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:14.540022 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:14.539643 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:14.540022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:14.539526 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:14.540022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:14.539508 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:14.540022 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:14.539749 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:14.540022 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:14.539944 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:16.111795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:16.111733 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:16.112259 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:16.111817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:16.112259 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.111885 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:16.112259 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.111932 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:16.112259 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.111958 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:24.111937515 +0000 UTC m=+17.106434489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:16.112259 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.111979 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:24.111969125 +0000 UTC m=+17.106466081 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:16.212927 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:16.212886 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:16.213111 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.213034 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:16.213111 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.213051 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:16.213111 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.213061 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:16.213284 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.213115 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:24.213098071 +0000 UTC m=+17.207595046 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:16.540261 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:16.539528 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:16.540261 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.539685 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:16.540261 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:16.540120 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:16.540261 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.540216 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:16.540618 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:16.540310 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:16.540618 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:16.540417 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:18.539886 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:18.539843 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:18.540498 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:18.539843 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:18.540498 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:18.539979 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:18.540498 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:18.540074 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:18.540498 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:18.539853 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:18.540498 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:18.540176 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:20.539213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:20.539179 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:20.539213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:20.539214 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:20.539624 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:20.539293 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:20.539624 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:20.539297 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:20.539624 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:20.539381 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:20.539624 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:20.539425 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:22.540296 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:22.540254 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:22.540296 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:22.540284 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:22.540809 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:22.540284 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:22.540809 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:22.540407 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:22.540809 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:22.540500 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:22.540809 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:22.540581 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:24.177168 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:24.177125 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:24.177641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:24.177186 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:24.177641 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.177271 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:24.177641 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.177295 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:24.177641 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.177348 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:40.177329257 +0000 UTC m=+33.171826229 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:24.177641 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.177365 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:40.177358793 +0000 UTC m=+33.171855749 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:24.277945 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:24.277902 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:24.278119 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.278089 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:24.278119 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.278113 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:24.278203 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.278123 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:24.278203 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.278174 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:40.278161142 +0000 UTC m=+33.272658095 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:24.539984 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:24.539883 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:24.539984 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:24.539930 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:24.540236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:24.539884 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:24.540236 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.540023 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:24.540236 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.540083 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:24.540236 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:24.540166 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:26.539559 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:26.539522 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:26.540110 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:26.539523 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:26.540110 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:26.539664 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:26.540110 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:26.539522 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:26.540110 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:26.539742 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:26.540110 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:26.539824 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:28.539972 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:28.539934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:28.540466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:28.539943 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:28.540466 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:28.540059 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:28.540466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:28.539943 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:28.540466 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:28.540129 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:28.540466 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:28.540266 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:29.764883 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.764848 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qn58b" event={"ID":"48a0d645-1915-42a5-8aea-648f94ea0190","Type":"ContainerStarted","Data":"9e33b88bde1d7d9db1a5890bfb3168592852078e584626f120add51cd0e03b35"} Jul 27 19:17:29.766325 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.766299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerStarted","Data":"b7f7df581acc63141070b3c79671642b7e8cedc7f6e1fe7bb880c0fb41d61750"} Jul 27 19:17:29.767773 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.767704 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8ff5w" event={"ID":"a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0","Type":"ContainerStarted","Data":"993778237e259ed4d4b69e0f620d5eeb339a0fa785b383d276234237824b1a4c"} Jul 27 19:17:29.769495 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.769473 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"74930318d77c8066a92aaeebbae4c83f6e9e74d5dd6c1551a45148d865608048"} Jul 27 19:17:29.769563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.769530 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"7377c67f87bcfe34f6c0881560bb128430ef7d444bd4f20fc4c83c6151ea8162"} Jul 27 19:17:29.771155 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.771129 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hnnwt" event={"ID":"18f93f18-ce46-4be6-8870-9bb9db422b11","Type":"ContainerStarted","Data":"e332d7bd23771e98fd2c21d809c75cd464cfc6cf7fca3735c4e2a48f69c75a95"} Jul 27 19:17:29.773592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.773558 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" event={"ID":"bc30b7bb-1847-4164-91b6-338f667bdf24","Type":"ContainerStarted","Data":"85b821e663f635155db9050da1a732e4623cc34602813057f8496b2d5676a89a"} Jul 27 19:17:29.775113 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.775089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4wrcm" event={"ID":"57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4","Type":"ContainerStarted","Data":"47708e2ed5e0c7b08322b13e600a49397da4f506c276301aea425c0cbdcadd14"} Jul 27 19:17:29.776995 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.776970 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" event={"ID":"d4fa595d-8a37-41fd-b022-91a1a0d94b85","Type":"ContainerStarted","Data":"7b78182db5841058d0db9fa17288db8c3999d15999b41d5bb60c17dc0d8e3210"} Jul 27 19:17:29.779223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.779198 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" event={"ID":"4dbb63acc39cffb9179e67a48cdbfda3","Type":"ContainerStarted","Data":"31fa821e729753d430e0e8cff03bcbf1fba8eed9cab6bd3ea1e6c9535dd39115"} Jul 27 19:17:29.780639 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.780617 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"22d9fa04e8305f63fcc41b14bd23fdc0","Type":"ContainerStarted","Data":"3c438569208e5e68c99bfe283f3c0765faed2aad2ba5c0c1d146b27e4656d071"} Jul 27 19:17:29.826188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.826102 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hnnwt" podStartSLOduration=1.649147927 podStartE2EDuration="21.826087113s" totalImagesPullingTime="20.176939186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:29.810851789 +0000 UTC m=+22.805348764" watchObservedRunningTime="2026-07-27 19:17:29.826087113 +0000 UTC m=+22.820584087" Jul 27 19:17:29.826409 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.826381 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-244.ec2.internal" podStartSLOduration=21.826374021 podStartE2EDuration="21.826374021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:29.826084091 +0000 UTC m=+22.820581067" watchObservedRunningTime="2026-07-27 19:17:29.826374021 +0000 UTC m=+22.820871035" Jul 27 19:17:29.847245 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.847184 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w7q6w" podStartSLOduration=2.555981543 podStartE2EDuration="22.847169381s" totalImagesPullingTime="20.291187838s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:29.846725142 +0000 UTC m=+22.841222118" watchObservedRunningTime="2026-07-27 19:17:29.847169381 +0000 UTC m=+22.841666356" Jul 27 19:17:29.900620 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.900566 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-4wrcm" podStartSLOduration=2.659466401 podStartE2EDuration="22.900549357s" totalImagesPullingTime="20.241082956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:29.867126739 +0000 UTC m=+22.861623715" watchObservedRunningTime="2026-07-27 19:17:29.900549357 +0000 UTC m=+22.895046331" Jul 27 19:17:29.928850 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.928638 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-8ff5w" podStartSLOduration=2.6375363739999997 podStartE2EDuration="22.928623106s" totalImagesPullingTime="20.291086732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:29.900271842 +0000 UTC m=+22.894768819" watchObservedRunningTime="2026-07-27 19:17:29.928623106 +0000 UTC m=+22.923120081" Jul 27 19:17:29.929101 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:29.929073 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qn58b" podStartSLOduration=2.7495410270000002 podStartE2EDuration="22.929063906s" totalImagesPullingTime="20.179522879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:29.927120555 +0000 UTC m=+22.921617508" watchObservedRunningTime="2026-07-27 19:17:29.929063906 +0000 UTC m=+22.923560881" Jul 27 19:17:30.539951 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.539910 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:30.539951 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.539934 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:30.540297 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.539932 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:30.540297 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:30.540044 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:30.540297 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:30.540122 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:30.540297 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:30.540208 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:30.784364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.784332 2566 generic.go:356] "Generic (PLEG): container finished" podID="6136915b-a655-4dc2-9d87-51ada7dc77a6" containerID="b7f7df581acc63141070b3c79671642b7e8cedc7f6e1fe7bb880c0fb41d61750" exitCode=0 Jul 27 19:17:30.784916 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.784428 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerDied","Data":"b7f7df581acc63141070b3c79671642b7e8cedc7f6e1fe7bb880c0fb41d61750"} Jul 27 19:17:30.787940 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.787904 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"8ccfaa6a513a08b50e83790397c0e3d0ec5ebc15576fbd8c166bff47fa9acae2"} Jul 27 19:17:30.788046 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.787991 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"167875be77cae6da156c4bd97e94f3e908b6da1356323eea5d517644b6ad1973"} Jul 27 19:17:30.788046 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.788036 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"1a30b51020c6ffb04c068599f64a0e368f748557e85d78c7e4f9906c2af1e120"} Jul 27 19:17:30.788149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.788079 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"dfafa1d807c5979b9454e47d186fe3791bd646d0e5968eb047eb6258a453d48d"} Jul 27 19:17:30.789796 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.789755 2566 generic.go:356] "Generic (PLEG): container finished" podID="22d9fa04e8305f63fcc41b14bd23fdc0" containerID="3c438569208e5e68c99bfe283f3c0765faed2aad2ba5c0c1d146b27e4656d071" exitCode=0 Jul 27 19:17:30.789889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.789830 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"22d9fa04e8305f63fcc41b14bd23fdc0","Type":"ContainerDied","Data":"3c438569208e5e68c99bfe283f3c0765faed2aad2ba5c0c1d146b27e4656d071"} Jul 27 19:17:30.789889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.789870 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" event={"ID":"22d9fa04e8305f63fcc41b14bd23fdc0","Type":"ContainerStarted","Data":"f8f7fb38fa4ed925ff1d6de71ff1bfb5aab789eb156936cb9d476faa421e879f"} Jul 27 19:17:30.791918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.791856 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6sgs4" event={"ID":"663cc638-8629-41a2-81f3-e8bc031b6bb9","Type":"ContainerStarted","Data":"fedc64d7934939647da7fce66cf06d1f08c78cd9940f18004c68c320cc11d1fd"} Jul 27 19:17:30.817816 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.817737 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6sgs4" podStartSLOduration=3.611107068 podStartE2EDuration="23.817720856s" totalImagesPullingTime="20.206613788s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:30.817102607 +0000 UTC m=+23.811599582" watchObservedRunningTime="2026-07-27 19:17:30.817720856 +0000 UTC m=+23.812217830" Jul 27 19:17:30.833390 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:30.833341 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-244.ec2.internal" podStartSLOduration=22.833324307 podStartE2EDuration="22.833324307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:30.833231372 +0000 UTC m=+23.827728348" watchObservedRunningTime="2026-07-27 19:17:30.833324307 +0000 UTC m=+23.827821282" Jul 27 19:17:31.244326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.244292 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:31.244942 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.244925 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:31.366027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.366003 2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 27 19:17:31.515568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.515409 2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-27T19:17:31.366021895Z","UUID":"4792dd3e-26bc-424d-a5f7-1880c78334c3","Handler":null,"Name":"","Endpoint":""} Jul 27 19:17:31.517907 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.517885 2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 27 19:17:31.518030 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.517933 2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 27 19:17:31.796123 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.796049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" event={"ID":"bc30b7bb-1847-4164-91b6-338f667bdf24","Type":"ContainerStarted","Data":"322ff8f1b9deb2f7581d87a7f4b75dca85bc2fb5a9a85b5573fb7e7d605d1fa9"} Jul 27 19:17:31.796780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.796388 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:31.796854 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:31.796831 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qn58b" Jul 27 19:17:32.539748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:32.539715 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:32.540139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:32.539717 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:32.540139 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:32.539857 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:32.540139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:32.539717 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:32.540139 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:32.539965 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:32.540139 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:32.540056 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:32.802259 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:32.802211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"4d5ece3b0f2637829b2dd0824e84850b27b0f66412ee6c2b1da5aa972794db69"} Jul 27 19:17:33.807582 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:33.807536 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" event={"ID":"bc30b7bb-1847-4164-91b6-338f667bdf24","Type":"ContainerStarted","Data":"0d3f82a517ae4f6e8847e3e8e0471bdc96b87449017a5516832cd83aced229db"} Jul 27 19:17:33.828130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:33.828065 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-977mv" podStartSLOduration=3.735098136 podStartE2EDuration="26.828043102s" totalImagesPullingTime="23.092944966s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:33.827408322 +0000 UTC m=+26.821905298" watchObservedRunningTime="2026-07-27 19:17:33.828043102 +0000 UTC m=+26.822540078" Jul 27 19:17:34.539580 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:34.539547 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:34.539783 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:34.539591 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:34.539783 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:34.539595 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:34.539783 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:34.539682 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:34.539953 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:34.539794 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:34.539953 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:34.539878 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:35.821962 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.821656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" event={"ID":"4f1f1c0f-5832-423d-9bb5-7701a6e87553","Type":"ContainerStarted","Data":"5fb34006d11342065764f5cbff7345b37bf783719b59ad14f6681a4378b46f75"} Jul 27 19:17:35.822732 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.822034 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:35.822732 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.822081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:35.822732 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.822120 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:35.839329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.839302 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:35.839496 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.839480 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:17:35.893038 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:35.892986 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" podStartSLOduration=8.191510028 podStartE2EDuration="28.892969967s" totalImagesPullingTime="20.701459939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:35.859273324 +0000 UTC m=+28.853770299" watchObservedRunningTime="2026-07-27 19:17:35.892969967 +0000 UTC m=+28.887466942" Jul 27 19:17:36.539573 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:36.539534 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:36.539779 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:36.539649 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:36.539779 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:36.539551 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:36.539779 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:36.539544 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:36.539779 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:36.539742 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:36.539979 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:36.539873 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:36.826576 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:36.826483 2566 generic.go:356] "Generic (PLEG): container finished" podID="6136915b-a655-4dc2-9d87-51ada7dc77a6" containerID="e95f664e65188078371f091536796d246622df1e34a604516af3d69d535efcd7" exitCode=0 Jul 27 19:17:36.826576 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:36.826533 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerDied","Data":"e95f664e65188078371f091536796d246622df1e34a604516af3d69d535efcd7"} Jul 27 19:17:37.843735 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:37.843522 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccflr"] Jul 27 19:17:37.844308 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:37.843836 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:37.844308 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:37.843932 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:37.844817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:37.844789 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnw5n"] Jul 27 19:17:37.844927 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:37.844897 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:37.845027 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:37.845002 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:37.845522 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:37.845487 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqrkg"] Jul 27 19:17:37.845613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:37.845595 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:37.845716 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:37.845693 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:38.833784 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:38.833731 2566 generic.go:356] "Generic (PLEG): container finished" podID="6136915b-a655-4dc2-9d87-51ada7dc77a6" containerID="24cda0ff039726dabb553659c221dd831e3589ba375626aa33d412b3ecec1f7e" exitCode=0 Jul 27 19:17:38.833977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:38.833797 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerDied","Data":"24cda0ff039726dabb553659c221dd831e3589ba375626aa33d412b3ecec1f7e"} Jul 27 19:17:39.543785 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:39.543599 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:39.543785 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:39.543646 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:39.543785 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:39.543732 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:39.543785 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:39.543783 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:39.544357 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:39.543855 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:39.544357 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:39.543972 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:40.192777 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:40.192714 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:40.192777 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:40.192780 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:40.192970 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.192872 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:40.192970 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.192939 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.19292411 +0000 UTC m=+65.187421063 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 27 19:17:40.192970 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.192873 2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:40.193073 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.193009 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret podName:5bfae84b-f178-4938-a7db-12bc33b72023 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.192995235 +0000 UTC m=+65.187492187 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret") pod "global-pull-secret-syncer-vnw5n" (UID: "5bfae84b-f178-4938-a7db-12bc33b72023") : object "kube-system"/"original-pull-secret" not registered Jul 27 19:17:40.293288 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:40.293184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:40.293439 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.293368 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 27 19:17:40.293439 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.293390 2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 27 19:17:40.293439 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.293401 2566 projected.go:196] Error preparing data for projected volume kube-api-access-t6jqv for pod openshift-network-diagnostics/network-check-target-ccflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:40.293535 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:40.293467 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv podName:a8cb4ddd-e372-4b2d-b06c-3b613d158a10 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.293454598 +0000 UTC m=+65.287951551 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-t6jqv" (UniqueName: "kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv") pod "network-check-target-ccflr" (UID: "a8cb4ddd-e372-4b2d-b06c-3b613d158a10") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 27 19:17:40.841041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:40.840857 2566 generic.go:356] "Generic (PLEG): container finished" podID="6136915b-a655-4dc2-9d87-51ada7dc77a6" containerID="86203cd4e3a95fbf366b67590ab760f7d6c4cd17e96a972aafb0401c7e1092e6" exitCode=0 Jul 27 19:17:40.841041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:40.840936 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerDied","Data":"86203cd4e3a95fbf366b67590ab760f7d6c4cd17e96a972aafb0401c7e1092e6"} Jul 27 19:17:41.539538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:41.539504 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:41.539741 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:41.539521 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:41.539741 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:41.539645 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:41.539741 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:41.539638 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-vnw5n" podUID="5bfae84b-f178-4938-a7db-12bc33b72023" Jul 27 19:17:41.539741 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:41.539719 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tqrkg" podUID="dfedf762-12a2-4461-999d-071931cd94c7" Jul 27 19:17:41.539994 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:41.539821 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ccflr" podUID="a8cb4ddd-e372-4b2d-b06c-3b613d158a10" Jul 27 19:17:42.288645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.288607 2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-244.ec2.internal" event="NodeReady" Jul 27 19:17:42.289147 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.288780 2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 27 19:17:42.333086 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.333052 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-68854874cd-kz9vj"] Jul 27 19:17:42.356530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.356178 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68854874cd-kz9vj"] Jul 27 19:17:42.356530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.356373 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.357037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.357003 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jbsfn"] Jul 27 19:17:42.358700 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.358673 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f5t9g\"" Jul 27 19:17:42.359631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.359363 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 27 19:17:42.359960 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.359835 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 27 19:17:42.360257 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.360237 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 27 19:17:42.367220 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.367192 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 27 19:17:42.372175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.372146 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4ljv4"] Jul 27 19:17:42.372529 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.372498 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.375130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.374915 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 27 19:17:42.375130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.374943 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 27 19:17:42.375330 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.375315 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m4wg8\"" Jul 27 19:17:42.390268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.390232 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jbsfn"] Jul 27 19:17:42.390462 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.390287 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4ljv4"] Jul 27 19:17:42.390462 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.390342 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.393045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.393018 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 27 19:17:42.393045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.393045 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 27 19:17:42.393341 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.393318 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 27 19:17:42.393453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.393397 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dlv9z\"" Jul 27 19:17:42.509810 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.509774 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-certificates\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510002 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.509852 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnwn6\" (UniqueName: \"kubernetes.io/projected/004999e4-6617-4cd9-9d50-f57ad32a8d42-kube-api-access-jnwn6\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.510002 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.509903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-installation-pull-secrets\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510002 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.509938 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510002 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.509969 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns5wz\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-kube-api-access-ns5wz\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510205 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510059 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.510205 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510087 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrxj\" (UniqueName: \"kubernetes.io/projected/faa6c9de-d2e1-4800-8057-e7f07488fbe3-kube-api-access-kcrxj\") pod \"ingress-canary-4ljv4\" (UID: \"faa6c9de-d2e1-4800-8057-e7f07488fbe3\") " pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.510205 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510168 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-trusted-ca\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510334 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510202 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faa6c9de-d2e1-4800-8057-e7f07488fbe3-cert\") pod \"ingress-canary-4ljv4\" (UID: \"faa6c9de-d2e1-4800-8057-e7f07488fbe3\") " pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.510334 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510258 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f40dd0b-02bb-4c1d-b406-43477032ce1b-ca-trust-extracted\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510334 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510295 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/004999e4-6617-4cd9-9d50-f57ad32a8d42-tmp-dir\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.510334 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-bound-sa-token\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510383 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-image-registry-private-configuration\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.510530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.510424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/004999e4-6617-4cd9-9d50-f57ad32a8d42-config-volume\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.611656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/004999e4-6617-4cd9-9d50-f57ad32a8d42-tmp-dir\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.611656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611613 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-bound-sa-token\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.611656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611641 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-image-registry-private-configuration\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.611656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611660 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/004999e4-6617-4cd9-9d50-f57ad32a8d42-config-volume\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611797 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-certificates\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnwn6\" (UniqueName: \"kubernetes.io/projected/004999e4-6617-4cd9-9d50-f57ad32a8d42-kube-api-access-jnwn6\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-installation-pull-secrets\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611895 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611920 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns5wz\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-kube-api-access-ns5wz\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611962 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.612014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.611984 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrxj\" (UniqueName: \"kubernetes.io/projected/faa6c9de-d2e1-4800-8057-e7f07488fbe3-kube-api-access-kcrxj\") pod \"ingress-canary-4ljv4\" (UID: \"faa6c9de-d2e1-4800-8057-e7f07488fbe3\") " pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.612374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-trusted-ca\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612041 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/004999e4-6617-4cd9-9d50-f57ad32a8d42-tmp-dir\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.612374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faa6c9de-d2e1-4800-8057-e7f07488fbe3-cert\") pod \"ingress-canary-4ljv4\" (UID: \"faa6c9de-d2e1-4800-8057-e7f07488fbe3\") " pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.612374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f40dd0b-02bb-4c1d-b406-43477032ce1b-ca-trust-extracted\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612307 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/004999e4-6617-4cd9-9d50-f57ad32a8d42-config-volume\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.612623 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612462 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f40dd0b-02bb-4c1d-b406-43477032ce1b-ca-trust-extracted\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612623 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.612484 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-certificates\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.612623 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:42.612575 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:17:42.612623 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:42.612599 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:17:42.612623 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:42.612602 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:17:42.612894 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:42.612655 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:43.112639042 +0000 UTC m=+36.107136005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:17:42.612894 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:42.612670 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:17:43.112662725 +0000 UTC m=+36.107159684 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:17:42.613401 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.613375 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-trusted-ca\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.617358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.617325 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-installation-pull-secrets\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.617358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.617348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/faa6c9de-d2e1-4800-8057-e7f07488fbe3-cert\") pod \"ingress-canary-4ljv4\" (UID: \"faa6c9de-d2e1-4800-8057-e7f07488fbe3\") " pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.617609 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.617349 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-image-registry-private-configuration\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.624733 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.624706 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-bound-sa-token\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.624925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.624795 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnwn6\" (UniqueName: \"kubernetes.io/projected/004999e4-6617-4cd9-9d50-f57ad32a8d42-kube-api-access-jnwn6\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:42.624925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.624856 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns5wz\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-kube-api-access-ns5wz\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:42.625808 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.625782 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrxj\" (UniqueName: \"kubernetes.io/projected/faa6c9de-d2e1-4800-8057-e7f07488fbe3-kube-api-access-kcrxj\") pod \"ingress-canary-4ljv4\" (UID: \"faa6c9de-d2e1-4800-8057-e7f07488fbe3\") " pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.701782 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.701731 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4ljv4" Jul 27 19:17:42.887023 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:42.886934 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4ljv4"] Jul 27 19:17:42.893589 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:17:42.893559 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaa6c9de_d2e1_4800_8057_e7f07488fbe3.slice/crio-fd4629825ae676203db745ed6328e225a54d9a9201c85104e128933750afbb27 WatchSource:0}: Error finding container fd4629825ae676203db745ed6328e225a54d9a9201c85104e128933750afbb27: Status 404 returned error can't find the container with id fd4629825ae676203db745ed6328e225a54d9a9201c85104e128933750afbb27 Jul 27 19:17:43.116860 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.116821 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:43.117034 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.116881 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:43.117034 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:43.116990 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:17:43.117123 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:43.117086 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:44.117063215 +0000 UTC m=+37.111560171 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:17:43.117171 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:43.116992 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:17:43.117171 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:43.117155 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:17:43.117246 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:43.117206 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:17:44.117193204 +0000 UTC m=+37.111690176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:17:43.540584 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.540117 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:17:43.540584 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.540321 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:17:43.540584 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.540120 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:17:43.543103 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.543079 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 19:17:43.543223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.543122 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 19:17:43.543902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.543876 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 19:17:43.544109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.544089 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 19:17:43.544109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.544126 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lvnq9\"" Jul 27 19:17:43.544398 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.544308 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d7dln\"" Jul 27 19:17:43.850677 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:43.850588 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4ljv4" event={"ID":"faa6c9de-d2e1-4800-8057-e7f07488fbe3","Type":"ContainerStarted","Data":"fd4629825ae676203db745ed6328e225a54d9a9201c85104e128933750afbb27"} Jul 27 19:17:44.126750 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:44.126668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:44.126750 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:44.126730 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:44.126990 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:44.126817 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:17:44.126990 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:44.126842 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:17:44.126990 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:44.126848 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:17:44.126990 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:44.126904 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:17:46.126884266 +0000 UTC m=+39.121381223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:17:44.126990 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:44.126921 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:46.126913241 +0000 UTC m=+39.121410196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:17:46.035884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.035678 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:46.146254 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.146165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:46.146254 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.146209 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:46.146438 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:46.146312 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:17:46.146438 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:46.146372 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:50.146358448 +0000 UTC m=+43.140855402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:17:46.146438 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:46.146313 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:17:46.146438 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:46.146410 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:17:46.146584 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:46.146455 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:17:50.146443235 +0000 UTC m=+43.140940190 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:17:46.171515 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.171481 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:46.271780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.271735 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:46.500399 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.500365 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:46.570249 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.570218 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:46.658309 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.658276 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:46.860686 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.860606 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4ljv4" event={"ID":"faa6c9de-d2e1-4800-8057-e7f07488fbe3","Type":"ContainerStarted","Data":"c3958aefbab548f6c78197033a81bf50f23cca6e1115745d000c4a846711fb63"} Jul 27 19:17:46.863176 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.863151 2566 generic.go:356] "Generic (PLEG): container finished" podID="6136915b-a655-4dc2-9d87-51ada7dc77a6" containerID="f45592a1644c30ad21a5a789ec971db8bb3e5c42532fb40a35596a8f53eeadf0" exitCode=0 Jul 27 19:17:46.863314 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.863200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerDied","Data":"f45592a1644c30ad21a5a789ec971db8bb3e5c42532fb40a35596a8f53eeadf0"} Jul 27 19:17:46.887867 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:46.887808 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4ljv4" podStartSLOduration=1.924540073 podStartE2EDuration="4.887752994s" totalImagesPullingTime="2.963212921s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:46.887351735 +0000 UTC m=+39.881848711" watchObservedRunningTime="2026-07-27 19:17:46.887752994 +0000 UTC m=+39.882249948" Jul 27 19:17:47.868479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:47.868439 2566 generic.go:356] "Generic (PLEG): container finished" podID="6136915b-a655-4dc2-9d87-51ada7dc77a6" containerID="6dcf9193dbedfa91b03ef57f232343f5aaf541defa3a80e770d052531fffbeb8" exitCode=0 Jul 27 19:17:47.868989 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:47.868529 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerDied","Data":"6dcf9193dbedfa91b03ef57f232343f5aaf541defa3a80e770d052531fffbeb8"} Jul 27 19:17:48.874079 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:48.874038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-j54z8" event={"ID":"6136915b-a655-4dc2-9d87-51ada7dc77a6","Type":"ContainerStarted","Data":"38fbf32a5c3345e024211542e2471a84c36b12630e9d87f778b4c5338c104f8c"} Jul 27 19:17:48.903222 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:48.903113 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-j54z8" podStartSLOduration=10.351355897 podStartE2EDuration="41.903088956s" totalImagesPullingTime="31.551733059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:17:48.900939838 +0000 UTC m=+41.895436813" watchObservedRunningTime="2026-07-27 19:17:48.903088956 +0000 UTC m=+41.897585932" Jul 27 19:17:49.036499 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:49.036440 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:49.141270 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:49.141175 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:49.211076 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:49.211043 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:50.175568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:50.175520 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:50.175568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:50.175574 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:50.176080 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:50.175676 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:17:50.176080 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:50.175691 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:17:50.176080 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:50.175714 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:17:50.176080 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:50.175725 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:17:58.175712431 +0000 UTC m=+51.170209384 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:17:50.176080 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:50.175787 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:17:58.175750981 +0000 UTC m=+51.170247938 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:17:55.056039 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:55.056000 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:55.157258 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:55.157219 2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" limit=1024 Jul 27 19:17:58.236426 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:58.236389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:17:58.236863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:17:58.236469 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:17:58.236863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:58.236561 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:17:58.236863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:58.236577 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:17:58.236863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:58.236588 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:17:58.236863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:58.236632 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:14.236614034 +0000 UTC m=+67.231110987 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:17:58.236863 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:17:58.236647 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:18:14.236641094 +0000 UTC m=+67.231138047 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:18:07.840151 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:07.840120 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-7t8np" Jul 27 19:18:11.646162 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.646127 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr"] Jul 27 19:18:11.650546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.650524 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" Jul 27 19:18:11.653175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.653135 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 27 19:18:11.653985 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.653967 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:18:11.653985 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.653978 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-2pfnk\"" Jul 27 19:18:11.663075 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.663046 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr"] Jul 27 19:18:11.733968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.733927 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbbkb\" (UniqueName: \"kubernetes.io/projected/febd5129-bfda-4f84-93cd-a00c74925745-kube-api-access-pbbkb\") pod \"volume-data-source-validator-549c6d54cf-v8tnr\" (UID: \"febd5129-bfda-4f84-93cd-a00c74925745\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" Jul 27 19:18:11.749826 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.749790 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-85766f5879-5lm2d"] Jul 27 19:18:11.752677 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.752651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.757102 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.757070 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4"] Jul 27 19:18:11.757554 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.757533 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 27 19:18:11.758037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.757794 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 27 19:18:11.758037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.757797 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 27 19:18:11.758483 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.758460 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 27 19:18:11.758679 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.758660 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-bv2gj\"" Jul 27 19:18:11.759153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.759136 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 27 19:18:11.759274 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.759211 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 27 19:18:11.776754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.776716 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85766f5879-5lm2d"] Jul 27 19:18:11.776945 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.776860 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.785785 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.785741 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 27 19:18:11.786000 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.785745 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-psmpm\"" Jul 27 19:18:11.786204 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.785828 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 27 19:18:11.786280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.785883 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 27 19:18:11.788228 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.788201 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 27 19:18:11.794093 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.794066 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bj8bq"] Jul 27 19:18:11.796973 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.796954 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6"] Jul 27 19:18:11.797091 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.797077 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.799791 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.799750 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" Jul 27 19:18:11.802259 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.802235 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 27 19:18:11.802259 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.802252 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-f2vr9\"" Jul 27 19:18:11.802442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.802238 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 27 19:18:11.802442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.802238 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:18:11.802672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.802657 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 27 19:18:11.805372 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.805349 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-rv5rm\"" Jul 27 19:18:11.806204 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.806184 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4"] Jul 27 19:18:11.817482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.817454 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6"] Jul 27 19:18:11.819507 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.819486 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 27 19:18:11.834717 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834673 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24qn\" (UniqueName: \"kubernetes.io/projected/6355fff2-3591-45c4-a7e3-d30721cf1b69-kube-api-access-z24qn\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.834900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834747 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.834900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834800 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b2df6ab-5abe-407a-a849-f0b37f84e426-trusted-ca\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.834900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834824 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-default-certificate\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.834900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834838 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b2df6ab-5abe-407a-a849-f0b37f84e426-config\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.834900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834862 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/76da00ea-e152-40a1-b5a3-1beedee179f4-kube-api-access-sfl8t\") pod \"network-check-source-7d88cb55d8-nvrx6\" (UID: \"76da00ea-e152-40a1-b5a3-1beedee179f4\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" Jul 27 19:18:11.834900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834896 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0232b815-0718-47fb-b4cb-7b709ac7bcdf-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.835085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834916 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.835085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.834984 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-stats-auth\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.835085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.835007 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzt76\" (UniqueName: \"kubernetes.io/projected/4b2df6ab-5abe-407a-a849-f0b37f84e426-kube-api-access-wzt76\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.835085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.835038 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.835085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.835080 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbbkb\" (UniqueName: \"kubernetes.io/projected/febd5129-bfda-4f84-93cd-a00c74925745-kube-api-access-pbbkb\") pod \"volume-data-source-validator-549c6d54cf-v8tnr\" (UID: \"febd5129-bfda-4f84-93cd-a00c74925745\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" Jul 27 19:18:11.835232 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.835102 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9t49\" (UniqueName: \"kubernetes.io/projected/0232b815-0718-47fb-b4cb-7b709ac7bcdf-kube-api-access-r9t49\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.835232 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.835119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2df6ab-5abe-407a-a849-f0b37f84e426-serving-cert\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.852939 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.852901 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bj8bq"] Jul 27 19:18:11.871808 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.871753 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbbkb\" (UniqueName: \"kubernetes.io/projected/febd5129-bfda-4f84-93cd-a00c74925745-kube-api-access-pbbkb\") pod \"volume-data-source-validator-549c6d54cf-v8tnr\" (UID: \"febd5129-bfda-4f84-93cd-a00c74925745\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" Jul 27 19:18:11.872290 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.872265 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-c8w4q"] Jul 27 19:18:11.875662 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.875641 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.878602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.878578 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 27 19:18:11.878726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.878580 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 27 19:18:11.878908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.878886 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 27 19:18:11.878973 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.878950 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 27 19:18:11.879601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.879583 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9dss8\"" Jul 27 19:18:11.885147 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.885122 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 27 19:18:11.905483 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.905408 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-c8w4q"] Jul 27 19:18:11.936064 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936024 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2df6ab-5abe-407a-a849-f0b37f84e426-serving-cert\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.936064 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936066 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24qn\" (UniqueName: \"kubernetes.io/projected/6355fff2-3591-45c4-a7e3-d30721cf1b69-kube-api-access-z24qn\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.936326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxhsp\" (UniqueName: \"kubernetes.io/projected/2bfb6007-ae94-4c6c-814f-eead25c3af21-kube-api-access-kxhsp\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.936326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.936326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b2df6ab-5abe-407a-a849-f0b37f84e426-trusted-ca\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.936326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936252 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-default-certificate\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.936326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b2df6ab-5abe-407a-a849-f0b37f84e426-config\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.936326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936309 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/76da00ea-e152-40a1-b5a3-1beedee179f4-kube-api-access-sfl8t\") pod \"network-check-source-7d88cb55d8-nvrx6\" (UID: \"76da00ea-e152-40a1-b5a3-1beedee179f4\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936336 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0232b815-0718-47fb-b4cb-7b709ac7bcdf-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936361 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfb6007-ae94-4c6c-814f-eead25c3af21-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936426 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bfb6007-ae94-4c6c-814f-eead25c3af21-snapshots\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936453 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfb6007-ae94-4c6c-814f-eead25c3af21-service-ca-bundle\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936493 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-stats-auth\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.936592 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936522 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzt76\" (UniqueName: \"kubernetes.io/projected/4b2df6ab-5abe-407a-a849-f0b37f84e426-kube-api-access-wzt76\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.936958 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:11.936282 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:11.936958 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:11.936552 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.436531478 +0000 UTC m=+65.431028451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : configmap references non-existent config key: service-ca.crt Jul 27 19:18:11.936958 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:11.936804 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls podName:0232b815-0718-47fb-b4cb-7b709ac7bcdf nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.436786266 +0000 UTC m=+65.431283225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5zpv4" (UID: "0232b815-0718-47fb-b4cb-7b709ac7bcdf") : secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:11.936958 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936842 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfb6007-ae94-4c6c-814f-eead25c3af21-tmp\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.936958 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936878 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfb6007-ae94-4c6c-814f-eead25c3af21-serving-cert\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:11.936958 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.936925 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.937275 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.937047 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/4b2df6ab-5abe-407a-a849-f0b37f84e426-config\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.937275 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:11.937156 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 19:18:11.937275 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:11.937215 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.437198229 +0000 UTC m=+65.431695186 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : secret "router-metrics-certs-default" not found Jul 27 19:18:11.937275 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.937063 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9t49\" (UniqueName: \"kubernetes.io/projected/0232b815-0718-47fb-b4cb-7b709ac7bcdf-kube-api-access-r9t49\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.937482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.937368 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4b2df6ab-5abe-407a-a849-f0b37f84e426-trusted-ca\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.937482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.937473 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/0232b815-0718-47fb-b4cb-7b709ac7bcdf-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.939128 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.939107 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-stats-auth\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.939128 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.939122 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-default-certificate\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.939251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.939124 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b2df6ab-5abe-407a-a849-f0b37f84e426-serving-cert\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.958370 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.958332 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24qn\" (UniqueName: \"kubernetes.io/projected/6355fff2-3591-45c4-a7e3-d30721cf1b69-kube-api-access-z24qn\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:11.958908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.958891 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" Jul 27 19:18:11.971879 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.971847 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzt76\" (UniqueName: \"kubernetes.io/projected/4b2df6ab-5abe-407a-a849-f0b37f84e426-kube-api-access-wzt76\") pod \"console-operator-584dcb6b54-bj8bq\" (UID: \"4b2df6ab-5abe-407a-a849-f0b37f84e426\") " pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:11.973440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.973419 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9t49\" (UniqueName: \"kubernetes.io/projected/0232b815-0718-47fb-b4cb-7b709ac7bcdf-kube-api-access-r9t49\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:11.976609 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.976584 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 27 19:18:11.992241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.992204 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 27 19:18:11.998398 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:11.998365 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfl8t\" (UniqueName: \"kubernetes.io/projected/76da00ea-e152-40a1-b5a3-1beedee179f4-kube-api-access-sfl8t\") pod \"network-check-source-7d88cb55d8-nvrx6\" (UID: \"76da00ea-e152-40a1-b5a3-1beedee179f4\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" Jul 27 19:18:12.034208 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.034164 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g"] Jul 27 19:18:12.038322 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038302 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.038726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038533 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfb6007-ae94-4c6c-814f-eead25c3af21-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.038726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038580 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bfb6007-ae94-4c6c-814f-eead25c3af21-snapshots\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.038726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038607 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfb6007-ae94-4c6c-814f-eead25c3af21-service-ca-bundle\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.038726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038649 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfb6007-ae94-4c6c-814f-eead25c3af21-tmp\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.038726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038685 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfb6007-ae94-4c6c-814f-eead25c3af21-serving-cert\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.039147 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.038942 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxhsp\" (UniqueName: \"kubernetes.io/projected/2bfb6007-ae94-4c6c-814f-eead25c3af21-kube-api-access-kxhsp\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.039311 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.039290 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2bfb6007-ae94-4c6c-814f-eead25c3af21-tmp\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.039583 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.039564 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2bfb6007-ae94-4c6c-814f-eead25c3af21-snapshots\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.039652 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.039591 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfb6007-ae94-4c6c-814f-eead25c3af21-service-ca-bundle\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.039744 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.039727 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2bfb6007-ae94-4c6c-814f-eead25c3af21-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.041567 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.041547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2bfb6007-ae94-4c6c-814f-eead25c3af21-serving-cert\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.051364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.051337 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:18:12.053823 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.053749 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 27 19:18:12.053823 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.053776 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 27 19:18:12.054375 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.054355 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-cdspv\"" Jul 27 19:18:12.054466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.054382 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 27 19:18:12.054736 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.054718 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-nz774"] Jul 27 19:18:12.057869 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.057850 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.058885 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.058864 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv"] Jul 27 19:18:12.062482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.062455 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.062622 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.062577 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g"] Jul 27 19:18:12.062932 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.062910 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 27 19:18:12.063348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.063328 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 27 19:18:12.063430 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.063366 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-s64st\"" Jul 27 19:18:12.064581 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.064566 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:18:12.069198 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.069175 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 27 19:18:12.069198 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.069190 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 27 19:18:12.069377 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.069213 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 27 19:18:12.069555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.069538 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-tqvpf\"" Jul 27 19:18:12.076534 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.076508 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-nz774"] Jul 27 19:18:12.077736 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.077715 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg"] Jul 27 19:18:12.081082 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.081054 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.081638 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.081616 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxhsp\" (UniqueName: \"kubernetes.io/projected/2bfb6007-ae94-4c6c-814f-eead25c3af21-kube-api-access-kxhsp\") pod \"insights-operator-5885fdb759-c8w4q\" (UID: \"2bfb6007-ae94-4c6c-814f-eead25c3af21\") " pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.084384 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.084365 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 27 19:18:12.084791 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.084753 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 27 19:18:12.085098 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.085082 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-t7255\"" Jul 27 19:18:12.085173 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.085082 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 27 19:18:12.104276 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.104073 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv"] Jul 27 19:18:12.104276 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.104139 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg"] Jul 27 19:18:12.107869 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.107841 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:12.112735 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.112712 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" Jul 27 19:18:12.119414 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.119318 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr"] Jul 27 19:18:12.122297 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.122268 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfebd5129_bfda_4f84_93cd_a00c74925745.slice/crio-245e5a86d9eea8865ea2ab023bb38809e77ab58b5fc98a3c2697768333c26326 WatchSource:0}: Error finding container 245e5a86d9eea8865ea2ab023bb38809e77ab58b5fc98a3c2697768333c26326: Status 404 returned error can't find the container with id 245e5a86d9eea8865ea2ab023bb38809e77ab58b5fc98a3c2697768333c26326 Jul 27 19:18:12.140445 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140396 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phm55\" (UniqueName: \"kubernetes.io/projected/cbb1d9dd-f50e-496f-baaa-51ea94028342-kube-api-access-phm55\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.140653 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2w65\" (UniqueName: \"kubernetes.io/projected/92aeab9f-cf67-4bc1-8337-892df9b815b0-kube-api-access-w2w65\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.140653 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140550 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.140653 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.140653 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140618 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.140653 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140643 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7bbk\" (UniqueName: \"kubernetes.io/projected/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-kube-api-access-v7bbk\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.140889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140730 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbb1d9dd-f50e-496f-baaa-51ea94028342-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.140889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbb1d9dd-f50e-496f-baaa-51ea94028342-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.140889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140835 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-config\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.141039 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.140996 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.185662 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.185405 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phm55\" (UniqueName: \"kubernetes.io/projected/cbb1d9dd-f50e-496f-baaa-51ea94028342-kube-api-access-phm55\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242631 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2w65\" (UniqueName: \"kubernetes.io/projected/92aeab9f-cf67-4bc1-8337-892df9b815b0-kube-api-access-w2w65\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242697 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242728 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242787 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7bbk\" (UniqueName: \"kubernetes.io/projected/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-kube-api-access-v7bbk\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242868 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbb1d9dd-f50e-496f-baaa-51ea94028342-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242918 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbb1d9dd-f50e-496f-baaa-51ea94028342-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.242977 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.243006 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-config\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.243672 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cbb1d9dd-f50e-496f-baaa-51ea94028342-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.243777 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 19:18:12.244268 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.243844 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert podName:0d1804d3-68c8-4a5e-acdd-cd30a74ee880 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.743823178 +0000 UTC m=+65.738320137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-nz774" (UID: "0d1804d3-68c8-4a5e-acdd-cd30a74ee880") : secret "networking-console-plugin-cert" not found Jul 27 19:18:12.245672 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.244005 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 19:18:12.245672 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.244044 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls podName:92aeab9f-cf67-4bc1-8337-892df9b815b0 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:12.744032517 +0000 UTC m=+65.738529484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls") pod "cluster-samples-operator-6748489656-2kwcg" (UID: "92aeab9f-cf67-4bc1-8337-892df9b815b0") : secret "samples-operator-tls" not found Jul 27 19:18:12.245672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.244715 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-config\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.245672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.245252 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.247261 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.246907 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cbb1d9dd-f50e-496f-baaa-51ea94028342-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.248374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.248352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-serving-cert\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.251896 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.251695 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 27 19:18:12.251896 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.251743 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 27 19:18:12.254520 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.254184 2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 27 19:18:12.254520 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.254322 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs podName:dfedf762-12a2-4461-999d-071931cd94c7 nodeName:}" failed. No retries permitted until 2026-07-27 19:19:16.254276089 +0000 UTC m=+129.248773059 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs") pod "network-metrics-daemon-tqrkg" (UID: "dfedf762-12a2-4461-999d-071931cd94c7") : secret "metrics-daemon-secret" not found Jul 27 19:18:12.256728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.256708 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfae84b-f178-4938-a7db-12bc33b72023-original-pull-secret\") pod \"global-pull-secret-syncer-vnw5n\" (UID: \"5bfae84b-f178-4938-a7db-12bc33b72023\") " pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:18:12.260308 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.260285 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2w65\" (UniqueName: \"kubernetes.io/projected/92aeab9f-cf67-4bc1-8337-892df9b815b0-kube-api-access-w2w65\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.267338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.267300 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-bj8bq"] Jul 27 19:18:12.269833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.269790 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7bbk\" (UniqueName: \"kubernetes.io/projected/fb04a0dd-b525-4d1b-ba6a-b5406e39737a-kube-api-access-v7bbk\") pod \"service-ca-operator-5d6c75f84f-qck7g\" (UID: \"fb04a0dd-b525-4d1b-ba6a-b5406e39737a\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.271170 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.271140 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b2df6ab_5abe_407a_a849_f0b37f84e426.slice/crio-4d40df2bd780f823c58eba67457899122f925cb047aeb093d0fc871eab351e79 WatchSource:0}: Error finding container 4d40df2bd780f823c58eba67457899122f925cb047aeb093d0fc871eab351e79: Status 404 returned error can't find the container with id 4d40df2bd780f823c58eba67457899122f925cb047aeb093d0fc871eab351e79 Jul 27 19:18:12.276230 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.276189 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phm55\" (UniqueName: \"kubernetes.io/projected/cbb1d9dd-f50e-496f-baaa-51ea94028342-kube-api-access-phm55\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-652sv\" (UID: \"cbb1d9dd-f50e-496f-baaa-51ea94028342\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.301036 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.301005 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6"] Jul 27 19:18:12.304265 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.304230 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76da00ea_e152_40a1_b5a3_1beedee179f4.slice/crio-ddba5dc1e61d8f5e2d3118f3558bacd6dc954b8d22349552021c45f127ecb735 WatchSource:0}: Error finding container ddba5dc1e61d8f5e2d3118f3558bacd6dc954b8d22349552021c45f127ecb735: Status 404 returned error can't find the container with id ddba5dc1e61d8f5e2d3118f3558bacd6dc954b8d22349552021c45f127ecb735 Jul 27 19:18:12.341898 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.341863 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-c8w4q"] Jul 27 19:18:12.344278 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.344245 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:18:12.347145 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.347123 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6jqv\" (UniqueName: \"kubernetes.io/projected/a8cb4ddd-e372-4b2d-b06c-3b613d158a10-kube-api-access-t6jqv\") pod \"network-check-target-ccflr\" (UID: \"a8cb4ddd-e372-4b2d-b06c-3b613d158a10\") " pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:18:12.348175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.348154 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" Jul 27 19:18:12.348475 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.348454 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfb6007_ae94_4c6c_814f_eead25c3af21.slice/crio-c20e23609d3c50974478fb04c8e86f90cc114624cce8e2cfb9eb699eed76855b WatchSource:0}: Error finding container c20e23609d3c50974478fb04c8e86f90cc114624cce8e2cfb9eb699eed76855b: Status 404 returned error can't find the container with id c20e23609d3c50974478fb04c8e86f90cc114624cce8e2cfb9eb699eed76855b Jul 27 19:18:12.359876 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.359853 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lvnq9\"" Jul 27 19:18:12.366548 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.366517 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:18:12.369836 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.369618 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-vnw5n" Jul 27 19:18:12.376532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.376509 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.445815 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.445877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.445960 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.445966 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.446034 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls podName:0232b815-0718-47fb-b4cb-7b709ac7bcdf nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.446012315 +0000 UTC m=+66.440509291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5zpv4" (UID: "0232b815-0718-47fb-b4cb-7b709ac7bcdf") : secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.446104 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.446153 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.446140848 +0000 UTC m=+66.440637809 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : secret "router-metrics-certs-default" not found Jul 27 19:18:12.446242 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.446214 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.446203195 +0000 UTC m=+66.440700148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : configmap references non-existent config key: service-ca.crt Jul 27 19:18:12.511416 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.511384 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g"] Jul 27 19:18:12.516725 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.516624 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb04a0dd_b525_4d1b_ba6a_b5406e39737a.slice/crio-46680d7a99a8deacc14bbc50719978e89f947e37fc038e86e7abea5298deab23 WatchSource:0}: Error finding container 46680d7a99a8deacc14bbc50719978e89f947e37fc038e86e7abea5298deab23: Status 404 returned error can't find the container with id 46680d7a99a8deacc14bbc50719978e89f947e37fc038e86e7abea5298deab23 Jul 27 19:18:12.544168 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.543483 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-vnw5n"] Jul 27 19:18:12.546942 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.546915 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfae84b_f178_4938_a7db_12bc33b72023.slice/crio-2525ecfc67e8200a585de865b198ca444712b0d798ac60e0cf36284619e275e2 WatchSource:0}: Error finding container 2525ecfc67e8200a585de865b198ca444712b0d798ac60e0cf36284619e275e2: Status 404 returned error can't find the container with id 2525ecfc67e8200a585de865b198ca444712b0d798ac60e0cf36284619e275e2 Jul 27 19:18:12.557585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.557550 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ccflr"] Jul 27 19:18:12.560947 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.560906 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8cb4ddd_e372_4b2d_b06c_3b613d158a10.slice/crio-9a6b02933655b753ae50a017c38a40d8ff67e09509ad73b9531bd7745f1b85e7 WatchSource:0}: Error finding container 9a6b02933655b753ae50a017c38a40d8ff67e09509ad73b9531bd7745f1b85e7: Status 404 returned error can't find the container with id 9a6b02933655b753ae50a017c38a40d8ff67e09509ad73b9531bd7745f1b85e7 Jul 27 19:18:12.576202 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.576172 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv"] Jul 27 19:18:12.578964 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:12.578939 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb1d9dd_f50e_496f_baaa_51ea94028342.slice/crio-dcb8fc93e005cb0db9f0d1385b61ab92f7cd402b1d70436b1b1924ef89f29c5b WatchSource:0}: Error finding container dcb8fc93e005cb0db9f0d1385b61ab92f7cd402b1d70436b1b1924ef89f29c5b: Status 404 returned error can't find the container with id dcb8fc93e005cb0db9f0d1385b61ab92f7cd402b1d70436b1b1924ef89f29c5b Jul 27 19:18:12.747788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.747672 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:12.747788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.747718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:12.748325 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.747847 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 19:18:12.748325 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.747856 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 19:18:12.748325 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.747900 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls podName:92aeab9f-cf67-4bc1-8337-892df9b815b0 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.747887099 +0000 UTC m=+66.742384052 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls") pod "cluster-samples-operator-6748489656-2kwcg" (UID: "92aeab9f-cf67-4bc1-8337-892df9b815b0") : secret "samples-operator-tls" not found Jul 27 19:18:12.748325 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:12.747914 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert podName:0d1804d3-68c8-4a5e-acdd-cd30a74ee880 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:13.747909035 +0000 UTC m=+66.742405989 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-nz774" (UID: "0d1804d3-68c8-4a5e-acdd-cd30a74ee880") : secret "networking-console-plugin-cert" not found Jul 27 19:18:12.931729 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.931690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" event={"ID":"febd5129-bfda-4f84-93cd-a00c74925745","Type":"ContainerStarted","Data":"245e5a86d9eea8865ea2ab023bb38809e77ab58b5fc98a3c2697768333c26326"} Jul 27 19:18:12.932977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.932942 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnw5n" event={"ID":"5bfae84b-f178-4938-a7db-12bc33b72023","Type":"ContainerStarted","Data":"2525ecfc67e8200a585de865b198ca444712b0d798ac60e0cf36284619e275e2"} Jul 27 19:18:12.934209 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.934185 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccflr" event={"ID":"a8cb4ddd-e372-4b2d-b06c-3b613d158a10","Type":"ContainerStarted","Data":"9a6b02933655b753ae50a017c38a40d8ff67e09509ad73b9531bd7745f1b85e7"} Jul 27 19:18:12.935463 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.935437 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" event={"ID":"4b2df6ab-5abe-407a-a849-f0b37f84e426","Type":"ContainerStarted","Data":"4d40df2bd780f823c58eba67457899122f925cb047aeb093d0fc871eab351e79"} Jul 27 19:18:12.936607 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.936566 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" event={"ID":"cbb1d9dd-f50e-496f-baaa-51ea94028342","Type":"ContainerStarted","Data":"dcb8fc93e005cb0db9f0d1385b61ab92f7cd402b1d70436b1b1924ef89f29c5b"} Jul 27 19:18:12.937708 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.937685 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" event={"ID":"2bfb6007-ae94-4c6c-814f-eead25c3af21","Type":"ContainerStarted","Data":"c20e23609d3c50974478fb04c8e86f90cc114624cce8e2cfb9eb699eed76855b"} Jul 27 19:18:12.939065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.939039 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" event={"ID":"76da00ea-e152-40a1-b5a3-1beedee179f4","Type":"ContainerStarted","Data":"ddba5dc1e61d8f5e2d3118f3558bacd6dc954b8d22349552021c45f127ecb735"} Jul 27 19:18:12.940201 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:12.940182 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" event={"ID":"fb04a0dd-b525-4d1b-ba6a-b5406e39737a","Type":"ContainerStarted","Data":"46680d7a99a8deacc14bbc50719978e89f947e37fc038e86e7abea5298deab23"} Jul 27 19:18:13.455823 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:13.455775 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:13.456000 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:13.455851 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:13.456000 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:13.455936 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:13.456234 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.456124 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 19:18:13.456234 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.456185 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:15.456166249 +0000 UTC m=+68.450663208 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : secret "router-metrics-certs-default" not found Jul 27 19:18:13.456345 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.456247 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:13.456345 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.456280 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls podName:0232b815-0718-47fb-b4cb-7b709ac7bcdf nodeName:}" failed. No retries permitted until 2026-07-27 19:18:15.456269636 +0000 UTC m=+68.450766595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5zpv4" (UID: "0232b815-0718-47fb-b4cb-7b709ac7bcdf") : secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:13.456448 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.456349 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:15.456338569 +0000 UTC m=+68.450835523 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : configmap references non-existent config key: service-ca.crt Jul 27 19:18:13.760172 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:13.759990 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:13.760172 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:13.760047 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:13.760798 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.760203 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 19:18:13.760798 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.760273 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert podName:0d1804d3-68c8-4a5e-acdd-cd30a74ee880 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:15.760252697 +0000 UTC m=+68.754749668 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-nz774" (UID: "0d1804d3-68c8-4a5e-acdd-cd30a74ee880") : secret "networking-console-plugin-cert" not found Jul 27 19:18:13.760798 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.760682 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 19:18:13.760798 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:13.760741 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls podName:92aeab9f-cf67-4bc1-8337-892df9b815b0 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:15.760722062 +0000 UTC m=+68.755219029 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls") pod "cluster-samples-operator-6748489656-2kwcg" (UID: "92aeab9f-cf67-4bc1-8337-892df9b815b0") : secret "samples-operator-tls" not found Jul 27 19:18:14.267721 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:14.266957 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:18:14.267721 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:14.267084 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:18:14.267721 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:14.267228 2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 27 19:18:14.267721 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:14.267300 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls podName:004999e4-6617-4cd9-9d50-f57ad32a8d42 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:46.267272916 +0000 UTC m=+99.261769887 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls") pod "dns-default-jbsfn" (UID: "004999e4-6617-4cd9-9d50-f57ad32a8d42") : secret "dns-default-metrics-tls" not found Jul 27 19:18:14.268361 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:14.268262 2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 27 19:18:14.268361 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:14.268285 2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-68854874cd-kz9vj: secret "image-registry-tls" not found Jul 27 19:18:14.268361 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:14.268338 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls podName:0f40dd0b-02bb-4c1d-b406-43477032ce1b nodeName:}" failed. No retries permitted until 2026-07-27 19:18:46.268322011 +0000 UTC m=+99.262818968 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls") pod "image-registry-68854874cd-kz9vj" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b") : secret "image-registry-tls" not found Jul 27 19:18:14.964324 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:14.963519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" event={"ID":"febd5129-bfda-4f84-93cd-a00c74925745","Type":"ContainerStarted","Data":"7132462edd347f505cc65240308575eaecc1f7bb39f3a8917936f5a6683c5db4"} Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:15.480686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:15.480774 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:15.480864 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.480907 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.480977 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls podName:0232b815-0718-47fb-b4cb-7b709ac7bcdf nodeName:}" failed. No retries permitted until 2026-07-27 19:18:19.480956492 +0000 UTC m=+72.475453468 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5zpv4" (UID: "0232b815-0718-47fb-b4cb-7b709ac7bcdf") : secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.480986 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.481018 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:19.481008268 +0000 UTC m=+72.475505237 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : secret "router-metrics-certs-default" not found Jul 27 19:18:15.481112 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.481082 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:19.481072152 +0000 UTC m=+72.475569120 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : configmap references non-existent config key: service-ca.crt Jul 27 19:18:15.784622 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:15.783584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:15.784622 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:15.783644 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:15.784622 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.783864 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 19:18:15.784622 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.783941 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls podName:92aeab9f-cf67-4bc1-8337-892df9b815b0 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:19.783917264 +0000 UTC m=+72.778414218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls") pod "cluster-samples-operator-6748489656-2kwcg" (UID: "92aeab9f-cf67-4bc1-8337-892df9b815b0") : secret "samples-operator-tls" not found Jul 27 19:18:15.785287 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.784014 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 19:18:15.785287 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:15.785223 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert podName:0d1804d3-68c8-4a5e-acdd-cd30a74ee880 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:19.785185439 +0000 UTC m=+72.779682405 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-nz774" (UID: "0d1804d3-68c8-4a5e-acdd-cd30a74ee880") : secret "networking-console-plugin-cert" not found Jul 27 19:18:19.523053 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:19.523010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:19.523081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:19.523166 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.523227 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.523263 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:27.523241464 +0000 UTC m=+80.517738418 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : configmap references non-existent config key: service-ca.crt Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.523302 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls podName:0232b815-0718-47fb-b4cb-7b709ac7bcdf nodeName:}" failed. No retries permitted until 2026-07-27 19:18:27.523275509 +0000 UTC m=+80.517772487 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5zpv4" (UID: "0232b815-0718-47fb-b4cb-7b709ac7bcdf") : secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.523345 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 19:18:19.523646 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.523402 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:27.523387518 +0000 UTC m=+80.517884484 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : secret "router-metrics-certs-default" not found Jul 27 19:18:19.827393 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:19.827297 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:19.827393 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:19.827365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:19.827637 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.827479 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 19:18:19.827637 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.827558 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls podName:92aeab9f-cf67-4bc1-8337-892df9b815b0 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:27.827538915 +0000 UTC m=+80.822035876 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls") pod "cluster-samples-operator-6748489656-2kwcg" (UID: "92aeab9f-cf67-4bc1-8337-892df9b815b0") : secret "samples-operator-tls" not found Jul 27 19:18:19.827637 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.827479 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 19:18:19.827637 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:19.827630 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert podName:0d1804d3-68c8-4a5e-acdd-cd30a74ee880 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:27.827612237 +0000 UTC m=+80.822109191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-nz774" (UID: "0d1804d3-68c8-4a5e-acdd-cd30a74ee880") : secret "networking-console-plugin-cert" not found Jul 27 19:18:20.984839 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.984801 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" event={"ID":"2bfb6007-ae94-4c6c-814f-eead25c3af21","Type":"ContainerStarted","Data":"370b03260ca3be8e519a474a9798df67293ef8944b3ce65608bab62402105874"} Jul 27 19:18:20.986558 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.986519 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" event={"ID":"76da00ea-e152-40a1-b5a3-1beedee179f4","Type":"ContainerStarted","Data":"2b687c910097e6e03e302843ad05f205dc3a0ab2bcb00f32b2df9f87baeb33a8"} Jul 27 19:18:20.988309 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.988254 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" event={"ID":"fb04a0dd-b525-4d1b-ba6a-b5406e39737a","Type":"ContainerStarted","Data":"bdcfb7addd3866a8c5aea8ba9280c98974eaeb861a6175eee2c278e7b08e83d2"} Jul 27 19:18:20.989959 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.989788 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-vnw5n" event={"ID":"5bfae84b-f178-4938-a7db-12bc33b72023","Type":"ContainerStarted","Data":"49d464733e83aefdbd0e63bd80cb3bd8b0030a1a406fe29a8e125f639306fb05"} Jul 27 19:18:20.991777 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.991740 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ccflr" event={"ID":"a8cb4ddd-e372-4b2d-b06c-3b613d158a10","Type":"ContainerStarted","Data":"753995d4b2cbf6f81c7b00bc17041ebebdbd536d9289acc33df0711e9c979c39"} Jul 27 19:18:20.992069 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.992053 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:18:20.993509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.993488 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/0.log" Jul 27 19:18:20.993623 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.993526 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b2df6ab-5abe-407a-a849-f0b37f84e426" containerID="cec59f0ae9ea54511c537b6c1e078933e85f40d9123c9ffdf9b99069366f4af9" exitCode=255 Jul 27 19:18:20.993623 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.993608 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" event={"ID":"4b2df6ab-5abe-407a-a849-f0b37f84e426","Type":"ContainerDied","Data":"cec59f0ae9ea54511c537b6c1e078933e85f40d9123c9ffdf9b99069366f4af9"} Jul 27 19:18:20.993900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.993878 2566 scope.go:117] "RemoveContainer" containerID="cec59f0ae9ea54511c537b6c1e078933e85f40d9123c9ffdf9b99069366f4af9" Jul 27 19:18:20.995236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:20.995214 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" event={"ID":"cbb1d9dd-f50e-496f-baaa-51ea94028342","Type":"ContainerStarted","Data":"9a5237d7652f310c2c6445c76661a2a7acf63cdf2988b2cedd42136109b3d36b"} Jul 27 19:18:21.013622 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.013562 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" podStartSLOduration=2.142487879 podStartE2EDuration="10.013542183s" totalImagesPullingTime="7.871054304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:21.013357732 +0000 UTC m=+74.007854721" watchObservedRunningTime="2026-07-27 19:18:21.013542183 +0000 UTC m=+74.008039159" Jul 27 19:18:21.013830 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.013788 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-v8tnr" podStartSLOduration=8.145516213 podStartE2EDuration="10.013782749s" totalImagesPullingTime="1.868266536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:14.987269467 +0000 UTC m=+67.981766442" watchObservedRunningTime="2026-07-27 19:18:21.013782749 +0000 UTC m=+74.008279717" Jul 27 19:18:21.048875 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.048812 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ccflr" podStartSLOduration=66.390674506 podStartE2EDuration="1m14.048790081s" totalImagesPullingTime="7.658115575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:21.048249134 +0000 UTC m=+74.042746109" watchObservedRunningTime="2026-07-27 19:18:21.048790081 +0000 UTC m=+74.043287059" Jul 27 19:18:21.087682 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.087557 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" podStartSLOduration=2.381721973 podStartE2EDuration="10.087540781s" totalImagesPullingTime="7.705818808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:21.08653015 +0000 UTC m=+74.081027127" watchObservedRunningTime="2026-07-27 19:18:21.087540781 +0000 UTC m=+74.082037755" Jul 27 19:18:21.137199 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.137135 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-vnw5n" podStartSLOduration=66.446937628 podStartE2EDuration="1m14.137116188s" totalImagesPullingTime="7.69017856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:21.115015376 +0000 UTC m=+74.109512353" watchObservedRunningTime="2026-07-27 19:18:21.137116188 +0000 UTC m=+74.131613167" Jul 27 19:18:21.156181 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.156088 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-nvrx6" podStartSLOduration=2.238146926 podStartE2EDuration="10.156071744s" totalImagesPullingTime="7.917924818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:21.155587039 +0000 UTC m=+74.150084016" watchObservedRunningTime="2026-07-27 19:18:21.156071744 +0000 UTC m=+74.150568718" Jul 27 19:18:21.179406 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.179348 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" podStartSLOduration=1.538117138 podStartE2EDuration="9.17932707s" totalImagesPullingTime="7.641209932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:21.178324967 +0000 UTC m=+74.172821942" watchObservedRunningTime="2026-07-27 19:18:21.17932707 +0000 UTC m=+74.173824049" Jul 27 19:18:21.955831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.955786 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qmvch"] Jul 27 19:18:21.959165 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.959136 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:21.966123 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.966097 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 27 19:18:21.966123 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.966111 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 27 19:18:21.966631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.966571 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-djng7\"" Jul 27 19:18:21.993610 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:21.993574 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qmvch"] Jul 27 19:18:22.000712 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.000683 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:18:22.001284 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.001263 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/0.log" Jul 27 19:18:22.001407 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.001306 2566 generic.go:356] "Generic (PLEG): container finished" podID="4b2df6ab-5abe-407a-a849-f0b37f84e426" containerID="deedcf0c79af0bc4992e3042039a1f6fb36bdee639c67e14dd54b4c8636d1d5a" exitCode=255 Jul 27 19:18:22.001468 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.001391 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" event={"ID":"4b2df6ab-5abe-407a-a849-f0b37f84e426","Type":"ContainerDied","Data":"deedcf0c79af0bc4992e3042039a1f6fb36bdee639c67e14dd54b4c8636d1d5a"} Jul 27 19:18:22.001512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.001475 2566 scope.go:117] "RemoveContainer" containerID="cec59f0ae9ea54511c537b6c1e078933e85f40d9123c9ffdf9b99069366f4af9" Jul 27 19:18:22.006861 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.006803 2566 scope.go:117] "RemoveContainer" containerID="deedcf0c79af0bc4992e3042039a1f6fb36bdee639c67e14dd54b4c8636d1d5a" Jul 27 19:18:22.008194 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.007502 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bj8bq_openshift-console-operator(4b2df6ab-5abe-407a-a849-f0b37f84e426)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" podUID="4b2df6ab-5abe-407a-a849-f0b37f84e426" Jul 27 19:18:22.051998 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.051962 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.051998 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.052008 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18301c03-eb72-4b76-ade3-cdb912c0b462-data-volume\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.052221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.052090 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.052221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.052138 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18301c03-eb72-4b76-ade3-cdb912c0b462-crio-socket\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.052221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.052161 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjv68\" (UniqueName: \"kubernetes.io/projected/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-api-access-gjv68\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.108860 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.108826 2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:22.109035 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.108918 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:22.153347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.153310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.153550 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.153365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18301c03-eb72-4b76-ade3-cdb912c0b462-data-volume\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.153550 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.153455 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 19:18:22.153550 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.153545 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:22.653523373 +0000 UTC m=+75.648020340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : configmap "kube-rbac-proxy" not found Jul 27 19:18:22.153707 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.153636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.153707 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.153668 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18301c03-eb72-4b76-ade3-cdb912c0b462-crio-socket\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.153835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.153705 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjv68\" (UniqueName: \"kubernetes.io/projected/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-api-access-gjv68\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.153835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.153710 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/18301c03-eb72-4b76-ade3-cdb912c0b462-data-volume\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.154114 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.154088 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/18301c03-eb72-4b76-ade3-cdb912c0b462-crio-socket\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.154295 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.154279 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 19:18:22.154466 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.154445 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:22.654422672 +0000 UTC m=+75.648919837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : secret "insights-runtime-extractor-tls" not found Jul 27 19:18:22.195981 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.195946 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjv68\" (UniqueName: \"kubernetes.io/projected/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-api-access-gjv68\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.660006 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.659968 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.660188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:22.660033 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:22.660188 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.660114 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 19:18:22.660188 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.660139 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 19:18:22.660307 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.660194 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:23.660178628 +0000 UTC m=+76.654675581 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : configmap "kube-rbac-proxy" not found Jul 27 19:18:22.660307 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:22.660210 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:23.660203043 +0000 UTC m=+76.654699996 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : secret "insights-runtime-extractor-tls" not found Jul 27 19:18:23.005514 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:23.005486 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:18:23.005977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:23.005874 2566 scope.go:117] "RemoveContainer" containerID="deedcf0c79af0bc4992e3042039a1f6fb36bdee639c67e14dd54b4c8636d1d5a" Jul 27 19:18:23.006045 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:23.006029 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bj8bq_openshift-console-operator(4b2df6ab-5abe-407a-a849-f0b37f84e426)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" podUID="4b2df6ab-5abe-407a-a849-f0b37f84e426" Jul 27 19:18:23.670044 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:23.670008 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:23.670248 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:23.670077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:23.670248 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:23.670147 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 19:18:23.670248 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:23.670200 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 19:18:23.670248 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:23.670226 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:25.670202461 +0000 UTC m=+78.664699432 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : configmap "kube-rbac-proxy" not found Jul 27 19:18:23.670388 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:23.670251 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:25.670242019 +0000 UTC m=+78.664738975 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : secret "insights-runtime-extractor-tls" not found Jul 27 19:18:23.753136 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:23.753109 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wrcm_57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4/dns-node-resolver/0.log" Jul 27 19:18:24.008859 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:24.008829 2566 scope.go:117] "RemoveContainer" containerID="deedcf0c79af0bc4992e3042039a1f6fb36bdee639c67e14dd54b4c8636d1d5a" Jul 27 19:18:24.009235 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:24.008987 2566 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-bj8bq_openshift-console-operator(4b2df6ab-5abe-407a-a849-f0b37f84e426)\"" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" podUID="4b2df6ab-5abe-407a-a849-f0b37f84e426" Jul 27 19:18:24.944606 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:24.944581 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hnnwt_18f93f18-ce46-4be6-8870-9bb9db422b11/node-ca/0.log" Jul 27 19:18:25.549043 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:25.549000 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4ljv4_faa6c9de-d2e1-4800-8057-e7f07488fbe3/serve-healthcheck-canary/0.log" Jul 27 19:18:25.687850 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:25.687805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:25.688035 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:25.687874 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:25.688035 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:25.687933 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 19:18:25.688035 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:25.687965 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 19:18:25.688035 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:25.688005 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:29.687990087 +0000 UTC m=+82.682487040 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : configmap "kube-rbac-proxy" not found Jul 27 19:18:25.688035 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:25.688018 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:29.688012747 +0000 UTC m=+82.682509700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : secret "insights-runtime-extractor-tls" not found Jul 27 19:18:27.607217 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:27.607160 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:27.607217 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:27.607231 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:27.607717 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:27.607308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:27.607717 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.607341 2566 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:27.607717 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.607352 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.607333542 +0000 UTC m=+96.601830510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : configmap references non-existent config key: service-ca.crt Jul 27 19:18:27.607717 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.607414 2566 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 27 19:18:27.607717 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.607462 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs podName:6355fff2-3591-45c4-a7e3-d30721cf1b69 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.607444315 +0000 UTC m=+96.601941268 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs") pod "router-default-85766f5879-5lm2d" (UID: "6355fff2-3591-45c4-a7e3-d30721cf1b69") : secret "router-metrics-certs-default" not found Jul 27 19:18:27.607717 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.607478 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls podName:0232b815-0718-47fb-b4cb-7b709ac7bcdf nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.607469001 +0000 UTC m=+96.601965959 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5zpv4" (UID: "0232b815-0718-47fb-b4cb-7b709ac7bcdf") : secret "cluster-monitoring-operator-tls" not found Jul 27 19:18:27.910239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:27.910124 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:27.910239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:27.910183 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:27.910435 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.910292 2566 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 27 19:18:27.910435 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.910352 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert podName:0d1804d3-68c8-4a5e-acdd-cd30a74ee880 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.910337071 +0000 UTC m=+96.904834024 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-nz774" (UID: "0d1804d3-68c8-4a5e-acdd-cd30a74ee880") : secret "networking-console-plugin-cert" not found Jul 27 19:18:27.910435 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.910374 2566 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 27 19:18:27.910435 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:27.910418 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls podName:92aeab9f-cf67-4bc1-8337-892df9b815b0 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:43.910402489 +0000 UTC m=+96.904899442 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls") pod "cluster-samples-operator-6748489656-2kwcg" (UID: "92aeab9f-cf67-4bc1-8337-892df9b815b0") : secret "samples-operator-tls" not found Jul 27 19:18:29.727399 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:29.727276 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:29.727788 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:29.727408 2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 27 19:18:29.727788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:29.727445 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:29.727788 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:29.727469 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:37.727455599 +0000 UTC m=+90.721952553 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : configmap "kube-rbac-proxy" not found Jul 27 19:18:29.727788 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:29.727538 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 19:18:29.727788 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:29.727583 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:37.72757244 +0000 UTC m=+90.722069393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : secret "insights-runtime-extractor-tls" not found Jul 27 19:18:37.799118 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:37.799072 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:37.799530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:37.799144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:37.799530 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:37.799296 2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 27 19:18:37.799530 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:18:37.799370 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls podName:18301c03-eb72-4b76-ade3-cdb912c0b462 nodeName:}" failed. No retries permitted until 2026-07-27 19:18:53.799354589 +0000 UTC m=+106.793851543 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qmvch" (UID: "18301c03-eb72-4b76-ade3-cdb912c0b462") : secret "insights-runtime-extractor-tls" not found Jul 27 19:18:37.799660 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:37.799636 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/18301c03-eb72-4b76-ade3-cdb912c0b462-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:38.539513 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:38.539484 2566 scope.go:117] "RemoveContainer" containerID="deedcf0c79af0bc4992e3042039a1f6fb36bdee639c67e14dd54b4c8636d1d5a" Jul 27 19:18:39.057976 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:39.057950 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:18:39.058441 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:39.058004 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" event={"ID":"4b2df6ab-5abe-407a-a849-f0b37f84e426","Type":"ContainerStarted","Data":"727dd65a32d3291e38befc3446c62be93fc887ae7be94a752da12372bbbe510e"} Jul 27 19:18:39.058441 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:39.058290 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:39.063232 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:39.063207 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" Jul 27 19:18:39.075851 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:39.075801 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-bj8bq" podStartSLOduration=20.128638836 podStartE2EDuration="28.075784921s" totalImagesPullingTime="7.947146085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:39.07456403 +0000 UTC m=+92.069061029" watchObservedRunningTime="2026-07-27 19:18:39.075784921 +0000 UTC m=+92.070281924" Jul 27 19:18:43.652925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.652887 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:43.653556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.653030 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:43.653556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.653128 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:43.653670 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.653615 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6355fff2-3591-45c4-a7e3-d30721cf1b69-service-ca-bundle\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:43.655260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.655234 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/0232b815-0718-47fb-b4cb-7b709ac7bcdf-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5zpv4\" (UID: \"0232b815-0718-47fb-b4cb-7b709ac7bcdf\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:43.655399 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.655347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6355fff2-3591-45c4-a7e3-d30721cf1b69-metrics-certs\") pod \"router-default-85766f5879-5lm2d\" (UID: \"6355fff2-3591-45c4-a7e3-d30721cf1b69\") " pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:43.887390 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.887352 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:43.895507 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.895484 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" Jul 27 19:18:43.955361 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.955306 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:43.955533 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.955384 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:43.959479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.959020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92aeab9f-cf67-4bc1-8337-892df9b815b0-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-2kwcg\" (UID: \"92aeab9f-cf67-4bc1-8337-892df9b815b0\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:43.960400 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:43.960348 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0d1804d3-68c8-4a5e-acdd-cd30a74ee880-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-nz774\" (UID: \"0d1804d3-68c8-4a5e-acdd-cd30a74ee880\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:44.032243 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.032217 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-85766f5879-5lm2d"] Jul 27 19:18:44.034744 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:44.034715 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6355fff2_3591_45c4_a7e3_d30721cf1b69.slice/crio-5ec262dc0f58b2b1dea8aac993334cf002b98a77339fb5e6c5a22b2152213709 WatchSource:0}: Error finding container 5ec262dc0f58b2b1dea8aac993334cf002b98a77339fb5e6c5a22b2152213709: Status 404 returned error can't find the container with id 5ec262dc0f58b2b1dea8aac993334cf002b98a77339fb5e6c5a22b2152213709 Jul 27 19:18:44.049365 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.049338 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4"] Jul 27 19:18:44.052094 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:44.052063 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0232b815_0718_47fb_b4cb_7b709ac7bcdf.slice/crio-9e57bae1ad3a47bb7cad840cbe4ba709e063676ca813502681025465183309fe WatchSource:0}: Error finding container 9e57bae1ad3a47bb7cad840cbe4ba709e063676ca813502681025465183309fe: Status 404 returned error can't find the container with id 9e57bae1ad3a47bb7cad840cbe4ba709e063676ca813502681025465183309fe Jul 27 19:18:44.074814 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.074750 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" event={"ID":"0232b815-0718-47fb-b4cb-7b709ac7bcdf","Type":"ContainerStarted","Data":"9e57bae1ad3a47bb7cad840cbe4ba709e063676ca813502681025465183309fe"} Jul 27 19:18:44.075994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.075959 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85766f5879-5lm2d" event={"ID":"6355fff2-3591-45c4-a7e3-d30721cf1b69","Type":"ContainerStarted","Data":"5ec262dc0f58b2b1dea8aac993334cf002b98a77339fb5e6c5a22b2152213709"} Jul 27 19:18:44.170046 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.169953 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" Jul 27 19:18:44.190289 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.190262 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" Jul 27 19:18:44.348363 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.348326 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-nz774"] Jul 27 19:18:44.352911 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:44.352886 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1804d3_68c8_4a5e_acdd_cd30a74ee880.slice/crio-c770a68f7fa034453e712c8cfac9f76b95a887702f007a2a5c6c993c0ab45686 WatchSource:0}: Error finding container c770a68f7fa034453e712c8cfac9f76b95a887702f007a2a5c6c993c0ab45686: Status 404 returned error can't find the container with id c770a68f7fa034453e712c8cfac9f76b95a887702f007a2a5c6c993c0ab45686 Jul 27 19:18:44.365838 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:44.365808 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg"] Jul 27 19:18:45.084691 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:45.084656 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-85766f5879-5lm2d" event={"ID":"6355fff2-3591-45c4-a7e3-d30721cf1b69","Type":"ContainerStarted","Data":"19634d66ee1b4a63350070542e563f20dcf653cfb879b8c7869991b97243e0e8"} Jul 27 19:18:45.087027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:45.086978 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" event={"ID":"0d1804d3-68c8-4a5e-acdd-cd30a74ee880","Type":"ContainerStarted","Data":"c770a68f7fa034453e712c8cfac9f76b95a887702f007a2a5c6c993c0ab45686"} Jul 27 19:18:45.088383 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:45.088354 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" event={"ID":"92aeab9f-cf67-4bc1-8337-892df9b815b0","Type":"ContainerStarted","Data":"99e6001b290e7e8517c9c74f6ee708562a174f9b17363f5476223680961fdc76"} Jul 27 19:18:45.888223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:45.888188 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:45.891034 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:45.891008 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:45.921244 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:45.921179 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-85766f5879-5lm2d" podStartSLOduration=34.921156475 podStartE2EDuration="34.921156475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:45.107866111 +0000 UTC m=+98.102363086" watchObservedRunningTime="2026-07-27 19:18:45.921156475 +0000 UTC m=+98.915653452" Jul 27 19:18:46.095134 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.095092 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:46.096382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.096357 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-85766f5879-5lm2d" Jul 27 19:18:46.277796 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.277748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:18:46.278009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.277850 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:18:46.280631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.280600 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/004999e4-6617-4cd9-9d50-f57ad32a8d42-metrics-tls\") pod \"dns-default-jbsfn\" (UID: \"004999e4-6617-4cd9-9d50-f57ad32a8d42\") " pod="openshift-dns/dns-default-jbsfn" Jul 27 19:18:46.280787 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.280673 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"image-registry-68854874cd-kz9vj\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:18:46.286602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.286551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-m4wg8\"" Jul 27 19:18:46.293815 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.293785 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jbsfn" Jul 27 19:18:46.575259 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.575179 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-f5t9g\"" Jul 27 19:18:46.583220 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.583193 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:18:46.798134 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.798100 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-68854874cd-kz9vj"] Jul 27 19:18:46.801176 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:46.801144 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f40dd0b_02bb_4c1d_b406_43477032ce1b.slice/crio-6de3929d9f67893ab80952a3181bb80175b4ebc581ff01998ff852e3790fa8ca WatchSource:0}: Error finding container 6de3929d9f67893ab80952a3181bb80175b4ebc581ff01998ff852e3790fa8ca: Status 404 returned error can't find the container with id 6de3929d9f67893ab80952a3181bb80175b4ebc581ff01998ff852e3790fa8ca Jul 27 19:18:46.810805 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:46.810776 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jbsfn"] Jul 27 19:18:46.815311 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:46.815260 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004999e4_6617_4cd9_9d50_f57ad32a8d42.slice/crio-2d2ca908f8d96fdcf2c1d5abf41d062f5bdd7ec56e8db2b6c04760a8718f30e6 WatchSource:0}: Error finding container 2d2ca908f8d96fdcf2c1d5abf41d062f5bdd7ec56e8db2b6c04760a8718f30e6: Status 404 returned error can't find the container with id 2d2ca908f8d96fdcf2c1d5abf41d062f5bdd7ec56e8db2b6c04760a8718f30e6 Jul 27 19:18:47.101661 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.101546 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" event={"ID":"92aeab9f-cf67-4bc1-8337-892df9b815b0","Type":"ContainerStarted","Data":"c07c767488e6a21d885fcdbc6a64cf40bc763f40f2980ab95b93e4e61e90ea64"} Jul 27 19:18:47.101661 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.101624 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" event={"ID":"92aeab9f-cf67-4bc1-8337-892df9b815b0","Type":"ContainerStarted","Data":"b7041b0a51fa351825d9e67971eb09d182fbee302d3959b30da896ddd337aff1"} Jul 27 19:18:47.103303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.103257 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" event={"ID":"0232b815-0718-47fb-b4cb-7b709ac7bcdf","Type":"ContainerStarted","Data":"40b25dbab0311bb24e06f809f83f2eb2289f662d360288d44b4e04795c3b6277"} Jul 27 19:18:47.105040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.105009 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" event={"ID":"0d1804d3-68c8-4a5e-acdd-cd30a74ee880","Type":"ContainerStarted","Data":"c040572b9a929255fb77d26303dc8e9bdb7fe3fcad867308adc0aab7e8fdf55e"} Jul 27 19:18:47.106733 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.106705 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" event={"ID":"0f40dd0b-02bb-4c1d-b406-43477032ce1b","Type":"ContainerStarted","Data":"9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814"} Jul 27 19:18:47.106882 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.106770 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" event={"ID":"0f40dd0b-02bb-4c1d-b406-43477032ce1b","Type":"ContainerStarted","Data":"6de3929d9f67893ab80952a3181bb80175b4ebc581ff01998ff852e3790fa8ca"} Jul 27 19:18:47.106882 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.106797 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:18:47.107994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.107968 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbsfn" event={"ID":"004999e4-6617-4cd9-9d50-f57ad32a8d42","Type":"ContainerStarted","Data":"2d2ca908f8d96fdcf2c1d5abf41d062f5bdd7ec56e8db2b6c04760a8718f30e6"} Jul 27 19:18:47.120012 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.119945 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-2kwcg" podStartSLOduration=32.894099604 podStartE2EDuration="35.119930114s" totalImagesPullingTime="2.22583051s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:47.119252209 +0000 UTC m=+100.113749181" watchObservedRunningTime="2026-07-27 19:18:47.119930114 +0000 UTC m=+100.114427088" Jul 27 19:18:47.142474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.142420 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" podStartSLOduration=99.142405047 podStartE2EDuration="1m39.142405047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:47.141878814 +0000 UTC m=+100.136375822" watchObservedRunningTime="2026-07-27 19:18:47.142405047 +0000 UTC m=+100.136902023" Jul 27 19:18:47.161804 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.161717 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5zpv4" podStartSLOduration=33.583372167 podStartE2EDuration="36.161695737s" totalImagesPullingTime="2.57832357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:47.160792384 +0000 UTC m=+100.155289360" watchObservedRunningTime="2026-07-27 19:18:47.161695737 +0000 UTC m=+100.156192713" Jul 27 19:18:47.182608 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:47.182551 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-nz774" podStartSLOduration=32.906146132 podStartE2EDuration="35.182527975s" totalImagesPullingTime="2.276381843s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:47.181818902 +0000 UTC m=+100.176315877" watchObservedRunningTime="2026-07-27 19:18:47.182527975 +0000 UTC m=+100.177024951" Jul 27 19:18:49.117228 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:49.117191 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbsfn" event={"ID":"004999e4-6617-4cd9-9d50-f57ad32a8d42","Type":"ContainerStarted","Data":"aca68c8193da8c59735dc19bc719f7e9f5d2ab2a9a79e5dd0bd85665e4167139"} Jul 27 19:18:49.117637 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:49.117272 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jbsfn" event={"ID":"004999e4-6617-4cd9-9d50-f57ad32a8d42","Type":"ContainerStarted","Data":"dc21513aa35633daf6afe1871bcbe583f979efc496bc3240dff333f8de037c69"} Jul 27 19:18:49.117637 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:49.117337 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jbsfn" Jul 27 19:18:49.135745 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:49.135684 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jbsfn" podStartSLOduration=65.577043655 podStartE2EDuration="1m7.135666932s" totalImagesPullingTime="1.558623277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:18:49.134545605 +0000 UTC m=+102.129042580" watchObservedRunningTime="2026-07-27 19:18:49.135666932 +0000 UTC m=+102.130163907" Jul 27 19:18:52.004784 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:52.004738 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ccflr" Jul 27 19:18:53.844287 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:53.844249 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:53.851058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:53.851015 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/18301c03-eb72-4b76-ade3-cdb912c0b462-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qmvch\" (UID: \"18301c03-eb72-4b76-ade3-cdb912c0b462\") " pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:54.069778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:54.069722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qmvch" Jul 27 19:18:54.238705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:54.238669 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qmvch"] Jul 27 19:18:54.241649 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:18:54.241622 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18301c03_eb72_4b76_ade3_cdb912c0b462.slice/crio-03313e27c1de0752c0b53fd2095ebc37d0d393379d973d1aba47f9d1a4a2e304 WatchSource:0}: Error finding container 03313e27c1de0752c0b53fd2095ebc37d0d393379d973d1aba47f9d1a4a2e304: Status 404 returned error can't find the container with id 03313e27c1de0752c0b53fd2095ebc37d0d393379d973d1aba47f9d1a4a2e304 Jul 27 19:18:55.139489 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:55.139444 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmvch" event={"ID":"18301c03-eb72-4b76-ade3-cdb912c0b462","Type":"ContainerStarted","Data":"03313e27c1de0752c0b53fd2095ebc37d0d393379d973d1aba47f9d1a4a2e304"} Jul 27 19:18:59.122367 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:18:59.122329 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jbsfn" Jul 27 19:19:00.161144 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:00.161094 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmvch" event={"ID":"18301c03-eb72-4b76-ade3-cdb912c0b462","Type":"ContainerStarted","Data":"ff31df5cf3e3c04b3c77f805dcfc5d7720851ad12cd12dc56e843ffc502e0fd9"} Jul 27 19:19:04.177344 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:04.177310 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmvch" event={"ID":"18301c03-eb72-4b76-ade3-cdb912c0b462","Type":"ContainerStarted","Data":"b390aaa79f6f860b4302c74287336a65c2669f2378588af4352f94a28f36ea70"} Jul 27 19:19:08.115859 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:08.115826 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:19:08.194664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:08.194615 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qmvch" event={"ID":"18301c03-eb72-4b76-ade3-cdb912c0b462","Type":"ContainerStarted","Data":"76068552c942fec404ffcca236ef27e72e16d97ecc537ffd34f3a35a7492408f"} Jul 27 19:19:08.218649 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:08.218593 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qmvch" podStartSLOduration=34.062781803 podStartE2EDuration="47.218574306s" totalImagesPullingTime="13.155792503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 19:18:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:08.217338672 +0000 UTC m=+121.211835682" watchObservedRunningTime="2026-07-27 19:19:08.218574306 +0000 UTC m=+121.213071278" Jul 27 19:19:12.695698 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.695661 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl"] Jul 27 19:19:12.699247 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.699228 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.702271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.702246 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 27 19:19:12.702443 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.702247 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 27 19:19:12.702594 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.702577 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 27 19:19:12.702667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.702605 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-gl2xf\"" Jul 27 19:19:12.713246 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.713216 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl"] Jul 27 19:19:12.745672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.745639 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn"] Jul 27 19:19:12.748927 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.748903 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.751669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.751642 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 27 19:19:12.751669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.751654 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 27 19:19:12.751884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.751830 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 27 19:19:12.751884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.751836 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xzfj7\"" Jul 27 19:19:12.760441 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.760403 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn"] Jul 27 19:19:12.770506 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.770473 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jk2tq"] Jul 27 19:19:12.773887 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.773863 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.776434 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.776408 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 27 19:19:12.776579 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.776466 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-4mgvm\"" Jul 27 19:19:12.776730 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.776716 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 27 19:19:12.776902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.776887 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 27 19:19:12.814306 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.814270 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.814510 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.814319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n546w\" (UniqueName: \"kubernetes.io/projected/9de62a0d-47ff-4f0e-9d87-88a50c813191-kube-api-access-n546w\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.814510 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.814421 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.814510 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.814475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9de62a0d-47ff-4f0e-9d87-88a50c813191-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.915792 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.915724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-tls\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.915792 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.915796 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpc6s\" (UniqueName: \"kubernetes.io/projected/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-kube-api-access-wpc6s\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.915842 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n546w\" (UniqueName: \"kubernetes.io/projected/9de62a0d-47ff-4f0e-9d87-88a50c813191-kube-api-access-n546w\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.916047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.915864 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-wtmp\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.915974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b19c24bb-275c-4170-8d55-28a215b870ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.916047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916005 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-textfile\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916063 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-metrics-client-ca\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b19c24bb-275c-4170-8d55-28a215b870ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.916226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916151 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.916226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916171 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.916226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916201 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.916475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916315 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.916475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916345 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9de62a0d-47ff-4f0e-9d87-88a50c813191-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.916475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916424 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-root\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916454 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-sys\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916687 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916486 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:12.916687 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916519 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.916687 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.916544 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-th7w7\" (UniqueName: \"kubernetes.io/projected/b19c24bb-275c-4170-8d55-28a215b870ea-kube-api-access-th7w7\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:12.916687 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:19:12.916616 2566 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 27 19:19:12.916687 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:19:12.916685 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-tls podName:9de62a0d-47ff-4f0e-9d87-88a50c813191 nodeName:}" failed. No retries permitted until 2026-07-27 19:19:13.41666514 +0000 UTC m=+126.411162095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-5fjrl" (UID: "9de62a0d-47ff-4f0e-9d87-88a50c813191") : secret "openshift-state-metrics-tls" not found Jul 27 19:19:12.917066 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.917046 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9de62a0d-47ff-4f0e-9d87-88a50c813191-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.918601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.918585 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:12.924656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:12.924630 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n546w\" (UniqueName: \"kubernetes.io/projected/9de62a0d-47ff-4f0e-9d87-88a50c813191-kube-api-access-n546w\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:13.017294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017193 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-root\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017234 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-sys\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017255 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017286 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-th7w7\" (UniqueName: \"kubernetes.io/projected/b19c24bb-275c-4170-8d55-28a215b870ea-kube-api-access-th7w7\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-tls\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017314 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-root\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017323 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpc6s\" (UniqueName: \"kubernetes.io/projected/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-kube-api-access-wpc6s\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017373 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-wtmp\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017324 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-sys\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017403 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b19c24bb-275c-4170-8d55-28a215b870ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-textfile\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017596 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-metrics-client-ca\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017610 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-wtmp\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017634 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b19c24bb-275c-4170-8d55-28a215b870ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.017667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017653 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017815 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b19c24bb-275c-4170-8d55-28a215b870ea-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017865 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-textfile\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017900 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017964 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.017992 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-accelerators-collector-config\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.018233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.018214 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-metrics-client-ca\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.018560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.018410 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b19c24bb-275c-4170-8d55-28a215b870ea-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.018785 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.018746 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.020337 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.020312 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.020436 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.020368 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b19c24bb-275c-4170-8d55-28a215b870ea-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.020436 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.020371 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-tls\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.020436 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.020371 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.026598 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.026569 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpc6s\" (UniqueName: \"kubernetes.io/projected/7b509d8e-af00-442e-a7ea-3c25bbd4c7c5-kube-api-access-wpc6s\") pod \"node-exporter-jk2tq\" (UID: \"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5\") " pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.026718 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.026639 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-th7w7\" (UniqueName: \"kubernetes.io/projected/b19c24bb-275c-4170-8d55-28a215b870ea-kube-api-access-th7w7\") pod \"kube-state-metrics-56d9f944b4-cltnn\" (UID: \"b19c24bb-275c-4170-8d55-28a215b870ea\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.058671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.058631 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" Jul 27 19:19:13.083589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.083550 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jk2tq" Jul 27 19:19:13.097521 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:19:13.097484 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b509d8e_af00_442e_a7ea_3c25bbd4c7c5.slice/crio-3641b920ccac76da86068f1241abf46e48c81b09ec6be593ff267c0bb7ebb35a WatchSource:0}: Error finding container 3641b920ccac76da86068f1241abf46e48c81b09ec6be593ff267c0bb7ebb35a: Status 404 returned error can't find the container with id 3641b920ccac76da86068f1241abf46e48c81b09ec6be593ff267c0bb7ebb35a Jul 27 19:19:13.201620 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.201586 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn"] Jul 27 19:19:13.205292 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:19:13.205253 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb19c24bb_275c_4170_8d55_28a215b870ea.slice/crio-89358279dd2ddeb7461be0b9601ec9b5c34260ccb4d1881ecd695d7f3e1b088b WatchSource:0}: Error finding container 89358279dd2ddeb7461be0b9601ec9b5c34260ccb4d1881ecd695d7f3e1b088b: Status 404 returned error can't find the container with id 89358279dd2ddeb7461be0b9601ec9b5c34260ccb4d1881ecd695d7f3e1b088b Jul 27 19:19:13.213664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.213630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk2tq" event={"ID":"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5","Type":"ContainerStarted","Data":"3641b920ccac76da86068f1241abf46e48c81b09ec6be593ff267c0bb7ebb35a"} Jul 27 19:19:13.217538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.217511 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" event={"ID":"b19c24bb-275c-4170-8d55-28a215b870ea","Type":"ContainerStarted","Data":"89358279dd2ddeb7461be0b9601ec9b5c34260ccb4d1881ecd695d7f3e1b088b"} Jul 27 19:19:13.421231 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.421148 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:13.423647 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.423613 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9de62a0d-47ff-4f0e-9d87-88a50c813191-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-5fjrl\" (UID: \"9de62a0d-47ff-4f0e-9d87-88a50c813191\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:13.608141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.608098 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" Jul 27 19:19:13.805512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.804913 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl"] Jul 27 19:19:13.808516 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:19:13.808477 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de62a0d_47ff_4f0e_9d87_88a50c813191.slice/crio-d8346b5fe84fe1d6eee0646227d02a01a481f8df57af0c067660789c4fd6361b WatchSource:0}: Error finding container d8346b5fe84fe1d6eee0646227d02a01a481f8df57af0c067660789c4fd6361b: Status 404 returned error can't find the container with id d8346b5fe84fe1d6eee0646227d02a01a481f8df57af0c067660789c4fd6361b Jul 27 19:19:13.839712 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.839677 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:19:13.850107 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.848472 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:13.851985 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.851955 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-dl47z\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852195 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852399 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852460 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852659 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852715 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852901 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.852930 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.853100 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 27 19:19:13.853281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.853151 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 27 19:19:13.866747 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:13.866450 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:19:14.026933 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.026793 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93d94812-6114-4d0c-a730-0da27e8c97eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.026933 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.026834 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.026933 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.026856 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.026955 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-web-config\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027054 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027086 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93d94812-6114-4d0c-a730-0da27e8c97eb-config-out\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027121 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027165 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93d94812-6114-4d0c-a730-0da27e8c97eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027211 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93d94812-6114-4d0c-a730-0da27e8c97eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027244 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027267 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzkpv\" (UniqueName: \"kubernetes.io/projected/93d94812-6114-4d0c-a730-0da27e8c97eb-kube-api-access-qzkpv\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027299 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93d94812-6114-4d0c-a730-0da27e8c97eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.027474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.027330 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128175 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93d94812-6114-4d0c-a730-0da27e8c97eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128220 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128240 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzkpv\" (UniqueName: \"kubernetes.io/projected/93d94812-6114-4d0c-a730-0da27e8c97eb-kube-api-access-qzkpv\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93d94812-6114-4d0c-a730-0da27e8c97eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128315 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93d94812-6114-4d0c-a730-0da27e8c97eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128342 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128364 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-web-config\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128442 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93d94812-6114-4d0c-a730-0da27e8c97eb-config-out\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.128528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128506 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.129041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.128548 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93d94812-6114-4d0c-a730-0da27e8c97eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.129267 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.129240 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/93d94812-6114-4d0c-a730-0da27e8c97eb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.129665 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.129549 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/93d94812-6114-4d0c-a730-0da27e8c97eb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.129966 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.129940 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/93d94812-6114-4d0c-a730-0da27e8c97eb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.133636 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.132934 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.133636 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.133174 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/93d94812-6114-4d0c-a730-0da27e8c97eb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.133636 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.133592 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.133903 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.133682 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-web-config\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.134528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.134131 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.134528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.134476 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-config-volume\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.135003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.134976 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.135209 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.135186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/93d94812-6114-4d0c-a730-0da27e8c97eb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.135334 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.135311 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/93d94812-6114-4d0c-a730-0da27e8c97eb-config-out\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.144974 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.144939 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzkpv\" (UniqueName: \"kubernetes.io/projected/93d94812-6114-4d0c-a730-0da27e8c97eb-kube-api-access-qzkpv\") pod \"alertmanager-main-0\" (UID: \"93d94812-6114-4d0c-a730-0da27e8c97eb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.175881 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.175746 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 27 19:19:14.226440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.226311 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" event={"ID":"9de62a0d-47ff-4f0e-9d87-88a50c813191","Type":"ContainerStarted","Data":"1c6e4469910907beda7c3c41a8eb3b9cbf2fc08faf4d0fdb01c6481385d13a1e"} Jul 27 19:19:14.226440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.226385 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" event={"ID":"9de62a0d-47ff-4f0e-9d87-88a50c813191","Type":"ContainerStarted","Data":"fefeed58c25c503cd0ea77bd5952452de28a9545cf57b505e12719a944a469d2"} Jul 27 19:19:14.226440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.226418 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" event={"ID":"9de62a0d-47ff-4f0e-9d87-88a50c813191","Type":"ContainerStarted","Data":"d8346b5fe84fe1d6eee0646227d02a01a481f8df57af0c067660789c4fd6361b"} Jul 27 19:19:14.713748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:14.713710 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 27 19:19:14.736928 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:19:14.736889 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d94812_6114_4d0c_a730_0da27e8c97eb.slice/crio-31a735bbb583238274586d21be90a96bb3af6e92add9d2b2b49a3a14d7e5defd WatchSource:0}: Error finding container 31a735bbb583238274586d21be90a96bb3af6e92add9d2b2b49a3a14d7e5defd: Status 404 returned error can't find the container with id 31a735bbb583238274586d21be90a96bb3af6e92add9d2b2b49a3a14d7e5defd Jul 27 19:19:15.232173 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.232137 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"31a735bbb583238274586d21be90a96bb3af6e92add9d2b2b49a3a14d7e5defd"} Jul 27 19:19:15.234358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.234329 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" event={"ID":"b19c24bb-275c-4170-8d55-28a215b870ea","Type":"ContainerStarted","Data":"6728274bf978af357e86e4cbfdb480059e182b7ff3b21d9c91a7e7278cea875f"} Jul 27 19:19:15.234499 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.234400 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" event={"ID":"b19c24bb-275c-4170-8d55-28a215b870ea","Type":"ContainerStarted","Data":"b50b32bcdfc498cfc78c613c4fafcb68379a83ad26a68424634bf6434a2cecab"} Jul 27 19:19:15.234499 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.234431 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" event={"ID":"b19c24bb-275c-4170-8d55-28a215b870ea","Type":"ContainerStarted","Data":"52df88bd1832fa854a1aa6a3ed0199376634838ba45e1c8e7591f64e332fbf34"} Jul 27 19:19:15.235906 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.235880 2566 generic.go:356] "Generic (PLEG): container finished" podID="7b509d8e-af00-442e-a7ea-3c25bbd4c7c5" containerID="fccf92282e0fa686657825eade5c1b5eac2686f8b71dae43c4b0267a46d659f6" exitCode=0 Jul 27 19:19:15.236027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.235939 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk2tq" event={"ID":"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5","Type":"ContainerDied","Data":"fccf92282e0fa686657825eade5c1b5eac2686f8b71dae43c4b0267a46d659f6"} Jul 27 19:19:15.264565 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:15.264513 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-cltnn" podStartSLOduration=1.8820137350000001 podStartE2EDuration="3.264496393s" totalImagesPullingTime="1.382482658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:15.262744494 +0000 UTC m=+128.257241468" watchObservedRunningTime="2026-07-27 19:19:15.264496393 +0000 UTC m=+128.258993376" Jul 27 19:19:16.241173 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.241079 2566 generic.go:356] "Generic (PLEG): container finished" podID="93d94812-6114-4d0c-a730-0da27e8c97eb" containerID="ed4b5f44c3e184a679a1cfef75c54a6561d8c93f8176032648e1791d216bcf0b" exitCode=0 Jul 27 19:19:16.241644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.241171 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerDied","Data":"ed4b5f44c3e184a679a1cfef75c54a6561d8c93f8176032648e1791d216bcf0b"} Jul 27 19:19:16.243474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.243447 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" event={"ID":"9de62a0d-47ff-4f0e-9d87-88a50c813191","Type":"ContainerStarted","Data":"98dd3e38a284bb61a4365550b35755edb6fd55958cc48e33055ae772e2c0f4d5"} Jul 27 19:19:16.245686 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.245660 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk2tq" event={"ID":"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5","Type":"ContainerStarted","Data":"ffb4c70b06c9ced97c814982546916a5a47c7817b0c1d0b0778e8b35000ddfee"} Jul 27 19:19:16.245817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.245707 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jk2tq" event={"ID":"7b509d8e-af00-442e-a7ea-3c25bbd4c7c5","Type":"ContainerStarted","Data":"72fe71687f9feaba0e102144ddbe73f186def1a006b8c2e2bc97f68496db3af6"} Jul 27 19:19:16.302332 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.302275 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jk2tq" podStartSLOduration=2.771197772 podStartE2EDuration="4.302257417s" totalImagesPullingTime="1.438355709s" totalInitContainerRuntime="92.703936ms" isStatefulPod=true podCreationTimestamp="2026-07-27 19:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:16.300595958 +0000 UTC m=+129.295092934" watchObservedRunningTime="2026-07-27 19:19:16.302257417 +0000 UTC m=+129.296754392" Jul 27 19:19:16.352919 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.352858 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-5fjrl" podStartSLOduration=3.294363289 podStartE2EDuration="4.352839253s" totalImagesPullingTime="1.058475964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:16.348395994 +0000 UTC m=+129.342892959" watchObservedRunningTime="2026-07-27 19:19:16.352839253 +0000 UTC m=+129.347336246" Jul 27 19:19:16.353459 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.353418 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:19:16.355750 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.355725 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dfedf762-12a2-4461-999d-071931cd94c7-metrics-certs\") pod \"network-metrics-daemon-tqrkg\" (UID: \"dfedf762-12a2-4461-999d-071931cd94c7\") " pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:19:16.566409 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.566321 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d7dln\"" Jul 27 19:19:16.573326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.573298 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tqrkg" Jul 27 19:19:16.728486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:16.728452 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tqrkg"] Jul 27 19:19:16.731842 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:19:16.731811 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfedf762_12a2_4461_999d_071931cd94c7.slice/crio-af06fbbebe9a33e45aa70185343a58c904dec80f49e95cf7379c7bad53e3ec01 WatchSource:0}: Error finding container af06fbbebe9a33e45aa70185343a58c904dec80f49e95cf7379c7bad53e3ec01: Status 404 returned error can't find the container with id af06fbbebe9a33e45aa70185343a58c904dec80f49e95cf7379c7bad53e3ec01 Jul 27 19:19:17.253398 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.253353 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqrkg" event={"ID":"dfedf762-12a2-4461-999d-071931cd94c7","Type":"ContainerStarted","Data":"af06fbbebe9a33e45aa70185343a58c904dec80f49e95cf7379c7bad53e3ec01"} Jul 27 19:19:17.497522 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.497483 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-74fdbbdcd9-h9bvb"] Jul 27 19:19:17.502692 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.502646 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.506129 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.506052 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bhwfx\"" Jul 27 19:19:17.506841 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.506811 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 19:19:17.506971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.506843 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 19:19:17.507269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.507248 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 19:19:17.508470 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.508447 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 19:19:17.509011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.508815 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 19:19:17.509011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.508817 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 19:19:17.509630 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.509578 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 19:19:17.519126 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.519063 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 19:19:17.519633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.519604 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fdbbdcd9-h9bvb"] Jul 27 19:19:17.566400 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566358 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-trusted-ca-bundle\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.566585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566405 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-oauth-serving-cert\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.566585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-config\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.566585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566540 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552hv\" (UniqueName: \"kubernetes.io/projected/9667f9b2-188a-4de8-96ff-bb9bdbd59979-kube-api-access-552hv\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.566585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566576 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-oauth-config\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.566828 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566612 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-serving-cert\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.566828 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.566635 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-service-ca\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667655 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-config\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667921 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667717 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552hv\" (UniqueName: \"kubernetes.io/projected/9667f9b2-188a-4de8-96ff-bb9bdbd59979-kube-api-access-552hv\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667921 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667754 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-oauth-config\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667921 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667805 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-serving-cert\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667921 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667830 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-service-ca\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667921 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667871 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-trusted-ca-bundle\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.667921 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.667896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-oauth-serving-cert\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.668544 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.668520 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-config\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.668686 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.668602 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-oauth-serving-cert\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.668985 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.668948 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-service-ca\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.669365 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.669331 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-trusted-ca-bundle\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.671369 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.671347 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-serving-cert\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.671479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.671364 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-oauth-config\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.679551 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.679521 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552hv\" (UniqueName: \"kubernetes.io/projected/9667f9b2-188a-4de8-96ff-bb9bdbd59979-kube-api-access-552hv\") pod \"console-74fdbbdcd9-h9bvb\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:17.816579 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:17.816474 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:18.754033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:18.753997 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74fdbbdcd9-h9bvb"] Jul 27 19:19:18.756251 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:19:18.756219 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9667f9b2_188a_4de8_96ff_bb9bdbd59979.slice/crio-2e17f51771515c6804abafe33b40ccfc925875ff5f8e6ba8da678ad152f84830 WatchSource:0}: Error finding container 2e17f51771515c6804abafe33b40ccfc925875ff5f8e6ba8da678ad152f84830: Status 404 returned error can't find the container with id 2e17f51771515c6804abafe33b40ccfc925875ff5f8e6ba8da678ad152f84830 Jul 27 19:19:19.271656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.271620 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqrkg" event={"ID":"dfedf762-12a2-4461-999d-071931cd94c7","Type":"ContainerStarted","Data":"aba6ec71d3fff1b2784754f90a328ae52ee37e71a9a567732b6f55f080457db4"} Jul 27 19:19:19.271867 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.271686 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tqrkg" event={"ID":"dfedf762-12a2-4461-999d-071931cd94c7","Type":"ContainerStarted","Data":"487cfdb4ea853777e47fd0557212baa3eb4633af0f63e33e5163f6efb8cdbcd6"} Jul 27 19:19:19.276995 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.276963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"051f96d7b3ad7f7c82b3757dee30d804cebd92bc7d1dfbcce3c5ad5877218621"} Jul 27 19:19:19.277140 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.277049 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"c4ee879dbb628fec6a2e0eb54f736b7e7b1a462eebad99d6801d75ac32778abe"} Jul 27 19:19:19.277140 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.277093 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"58a6fe53c040806a0036d3f9dd15342e6c82324e8d656e4b50cbff38a9bf7b1e"} Jul 27 19:19:19.277140 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.277134 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"a8041878d9f11ff22a73f10d9693711e2d9dd27a0b2b7a195262994a4bf87f2b"} Jul 27 19:19:19.277251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.277204 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"91cab943b34bb4443bc861eb3be82a73add5d2e02613b8ebf27dac444124400e"} Jul 27 19:19:19.279080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.279043 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fdbbdcd9-h9bvb" event={"ID":"9667f9b2-188a-4de8-96ff-bb9bdbd59979","Type":"ContainerStarted","Data":"2e17f51771515c6804abafe33b40ccfc925875ff5f8e6ba8da678ad152f84830"} Jul 27 19:19:19.293169 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:19.292879 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tqrkg" podStartSLOduration=130.431362425 podStartE2EDuration="2m12.292856188s" totalImagesPullingTime="1.861493763s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:19.292588116 +0000 UTC m=+132.287085090" watchObservedRunningTime="2026-07-27 19:19:19.292856188 +0000 UTC m=+132.287353165" Jul 27 19:19:20.287457 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:20.287414 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"93d94812-6114-4d0c-a730-0da27e8c97eb","Type":"ContainerStarted","Data":"de66b1c1358881413ac61511d241cc056caee8fce8f571a75837316dd2cf82ce"} Jul 27 19:19:20.317562 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:20.317442 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.980615524 podStartE2EDuration="7.317418819s" totalImagesPullingTime="4.316750293s" totalInitContainerRuntime="20.053002ms" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:20.315833844 +0000 UTC m=+133.310330822" watchObservedRunningTime="2026-07-27 19:19:20.317418819 +0000 UTC m=+133.311915795" Jul 27 19:19:22.298935 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:22.298898 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fdbbdcd9-h9bvb" event={"ID":"9667f9b2-188a-4de8-96ff-bb9bdbd59979","Type":"ContainerStarted","Data":"ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade"} Jul 27 19:19:22.319670 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:22.319609 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-74fdbbdcd9-h9bvb" podStartSLOduration=2.497811424 podStartE2EDuration="5.319589062s" totalImagesPullingTime="2.821777638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:19:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:19:22.319261596 +0000 UTC m=+135.313758572" watchObservedRunningTime="2026-07-27 19:19:22.319589062 +0000 UTC m=+135.314086041" Jul 27 19:19:22.498178 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:22.498130 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68854874cd-kz9vj"] Jul 27 19:19:26.316346 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:26.316305 2566 generic.go:356] "Generic (PLEG): container finished" podID="cbb1d9dd-f50e-496f-baaa-51ea94028342" containerID="9a5237d7652f310c2c6445c76661a2a7acf63cdf2988b2cedd42136109b3d36b" exitCode=0 Jul 27 19:19:26.316825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:26.316371 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" event={"ID":"cbb1d9dd-f50e-496f-baaa-51ea94028342","Type":"ContainerDied","Data":"9a5237d7652f310c2c6445c76661a2a7acf63cdf2988b2cedd42136109b3d36b"} Jul 27 19:19:26.316825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:26.316694 2566 scope.go:117] "RemoveContainer" containerID="9a5237d7652f310c2c6445c76661a2a7acf63cdf2988b2cedd42136109b3d36b" Jul 27 19:19:27.322641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:27.322605 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-652sv" event={"ID":"cbb1d9dd-f50e-496f-baaa-51ea94028342","Type":"ContainerStarted","Data":"3a02ec77ddc369d4eae466fef81d9f66d9e2f96dba089567299e8945bb2ff24e"} Jul 27 19:19:27.816917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:27.816873 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:27.817120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:27.816941 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:27.822680 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:27.822649 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:28.330095 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:28.330063 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:19:47.518159 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.518046 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" podUID="0f40dd0b-02bb-4c1d-b406-43477032ce1b" containerName="registry" containerID="cri-o://9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814" gracePeriod=30 Jul 27 19:19:47.774339 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.774273 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:19:47.856033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.855988 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-installation-pull-secrets\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856035 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856286 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856078 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-certificates\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856286 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856099 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-trusted-ca\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856286 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856128 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns5wz\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-kube-api-access-ns5wz\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856286 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856155 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-image-registry-private-configuration\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856286 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856187 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f40dd0b-02bb-4c1d-b406-43477032ce1b-ca-trust-extracted\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856514 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856313 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-bound-sa-token\") pod \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\" (UID: \"0f40dd0b-02bb-4c1d-b406-43477032ce1b\") " Jul 27 19:19:47.856649 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856616 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:19:47.856649 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.856628 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:19:47.858884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.858848 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:19:47.859043 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.858859 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:19:47.859108 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.859079 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:19:47.859148 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.859122 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:19:47.859148 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.859130 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-kube-api-access-ns5wz" (OuterVolumeSpecName: "kube-api-access-ns5wz") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "kube-api-access-ns5wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:19:47.868009 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.867964 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f40dd0b-02bb-4c1d-b406-43477032ce1b-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0f40dd0b-02bb-4c1d-b406-43477032ce1b" (UID: "0f40dd0b-02bb-4c1d-b406-43477032ce1b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:19:47.957214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957178 2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-bound-sa-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957208 2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-installation-pull-secrets\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957219 2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957228 2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-registry-certificates\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957237 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0f40dd0b-02bb-4c1d-b406-43477032ce1b-trusted-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957246 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns5wz\" (UniqueName: \"kubernetes.io/projected/0f40dd0b-02bb-4c1d-b406-43477032ce1b-kube-api-access-ns5wz\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957255 2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0f40dd0b-02bb-4c1d-b406-43477032ce1b-image-registry-private-configuration\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:47.957479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:47.957266 2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0f40dd0b-02bb-4c1d-b406-43477032ce1b-ca-trust-extracted\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:19:48.405149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.405106 2566 generic.go:356] "Generic (PLEG): container finished" podID="0f40dd0b-02bb-4c1d-b406-43477032ce1b" containerID="9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814" exitCode=0 Jul 27 19:19:48.405323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.405154 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" Jul 27 19:19:48.405323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.405162 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" event={"ID":"0f40dd0b-02bb-4c1d-b406-43477032ce1b","Type":"ContainerDied","Data":"9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814"} Jul 27 19:19:48.405323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.405211 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-68854874cd-kz9vj" event={"ID":"0f40dd0b-02bb-4c1d-b406-43477032ce1b","Type":"ContainerDied","Data":"6de3929d9f67893ab80952a3181bb80175b4ebc581ff01998ff852e3790fa8ca"} Jul 27 19:19:48.405323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.405238 2566 scope.go:117] "RemoveContainer" containerID="9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814" Jul 27 19:19:48.414290 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.414267 2566 scope.go:117] "RemoveContainer" containerID="9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814" Jul 27 19:19:48.414628 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:19:48.414607 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814\": container with ID starting with 9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814 not found: ID does not exist" containerID="9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814" Jul 27 19:19:48.414667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.414637 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814"} err="failed to get container status \"9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814\": rpc error: code = NotFound desc = could not find container \"9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814\": container with ID starting with 9e058ef9c131747fab16a22dd1c7f4095bdcfbcea00171348362110d336c3814 not found: ID does not exist" Jul 27 19:19:48.432355 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.432320 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-68854874cd-kz9vj"] Jul 27 19:19:48.437420 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:48.437389 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-68854874cd-kz9vj"] Jul 27 19:19:49.544828 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:49.544791 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f40dd0b-02bb-4c1d-b406-43477032ce1b" path="/var/lib/kubelet/pods/0f40dd0b-02bb-4c1d-b406-43477032ce1b/volumes" Jul 27 19:19:51.426153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:51.426122 2566 generic.go:356] "Generic (PLEG): container finished" podID="fb04a0dd-b525-4d1b-ba6a-b5406e39737a" containerID="bdcfb7addd3866a8c5aea8ba9280c98974eaeb861a6175eee2c278e7b08e83d2" exitCode=0 Jul 27 19:19:51.426600 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:51.426192 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" event={"ID":"fb04a0dd-b525-4d1b-ba6a-b5406e39737a","Type":"ContainerDied","Data":"bdcfb7addd3866a8c5aea8ba9280c98974eaeb861a6175eee2c278e7b08e83d2"} Jul 27 19:19:51.426600 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:51.426554 2566 scope.go:117] "RemoveContainer" containerID="bdcfb7addd3866a8c5aea8ba9280c98974eaeb861a6175eee2c278e7b08e83d2" Jul 27 19:19:51.427930 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:51.427904 2566 generic.go:356] "Generic (PLEG): container finished" podID="2bfb6007-ae94-4c6c-814f-eead25c3af21" containerID="370b03260ca3be8e519a474a9798df67293ef8944b3ce65608bab62402105874" exitCode=0 Jul 27 19:19:51.428028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:51.427972 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" event={"ID":"2bfb6007-ae94-4c6c-814f-eead25c3af21","Type":"ContainerDied","Data":"370b03260ca3be8e519a474a9798df67293ef8944b3ce65608bab62402105874"} Jul 27 19:19:51.428308 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:51.428291 2566 scope.go:117] "RemoveContainer" containerID="370b03260ca3be8e519a474a9798df67293ef8944b3ce65608bab62402105874" Jul 27 19:19:52.439781 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:52.436818 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-qck7g" event={"ID":"fb04a0dd-b525-4d1b-ba6a-b5406e39737a","Type":"ContainerStarted","Data":"994f6e741dacfd5ffd814a3e372a1fec5ce874b149bb7b16547322f71a8467a0"} Jul 27 19:19:52.440229 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:19:52.440037 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-c8w4q" event={"ID":"2bfb6007-ae94-4c6c-814f-eead25c3af21","Type":"ContainerStarted","Data":"299b3aaed58ba6c1a46ac49280b3f84afb96e1266d1ad5084b6fa98a1942b951"} Jul 27 19:20:37.400853 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.400817 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-84c5c8b468-tdcsk"] Jul 27 19:20:37.401343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.401122 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f40dd0b-02bb-4c1d-b406-43477032ce1b" containerName="registry" Jul 27 19:20:37.401343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.401132 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f40dd0b-02bb-4c1d-b406-43477032ce1b" containerName="registry" Jul 27 19:20:37.401343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.401207 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f40dd0b-02bb-4c1d-b406-43477032ce1b" containerName="registry" Jul 27 19:20:37.404344 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.404326 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.407906 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.407883 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 27 19:20:37.408225 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.408207 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 27 19:20:37.408712 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.408690 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 27 19:20:37.408820 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.408696 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 27 19:20:37.408820 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.408730 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wlnjs\"" Jul 27 19:20:37.410586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.410565 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 27 19:20:37.415800 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.415752 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 27 19:20:37.423890 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.423862 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84c5c8b468-tdcsk"] Jul 27 19:20:37.485445 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485402 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-metrics-client-ca\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485445 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485449 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485480 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhj6\" (UniqueName: \"kubernetes.io/projected/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-kube-api-access-jxhj6\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485505 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-secret-telemeter-client\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485536 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485598 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-serving-certs-ca-bundle\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485651 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-telemeter-client-tls\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.485863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.485685 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-federate-client-tls\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.586993 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.586937 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-telemeter-client-tls\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.586993 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-federate-client-tls\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-metrics-client-ca\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587062 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587081 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhj6\" (UniqueName: \"kubernetes.io/projected/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-kube-api-access-jxhj6\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587098 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-secret-telemeter-client\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587120 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587159 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-serving-certs-ca-bundle\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.587946 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587916 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-metrics-client-ca\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.588140 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.587955 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-serving-certs-ca-bundle\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.588456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.588431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-telemeter-trusted-ca-bundle\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.589656 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.589633 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-telemeter-client-tls\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.589861 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.589836 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-secret-telemeter-client\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.589964 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.589947 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.590052 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.590035 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-federate-client-tls\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.603394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.603361 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhj6\" (UniqueName: \"kubernetes.io/projected/4d22b0b9-363a-4de2-aa06-6ddddf4bae90-kube-api-access-jxhj6\") pod \"telemeter-client-84c5c8b468-tdcsk\" (UID: \"4d22b0b9-363a-4de2-aa06-6ddddf4bae90\") " pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.714408 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.714361 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" Jul 27 19:20:37.875639 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:37.875606 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-84c5c8b468-tdcsk"] Jul 27 19:20:37.878840 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:20:37.878809 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d22b0b9_363a_4de2_aa06_6ddddf4bae90.slice/crio-c786f6efc4de026639410e4bedec9468a729c65e1250beb6fd42a038800d6b07 WatchSource:0}: Error finding container c786f6efc4de026639410e4bedec9468a729c65e1250beb6fd42a038800d6b07: Status 404 returned error can't find the container with id c786f6efc4de026639410e4bedec9468a729c65e1250beb6fd42a038800d6b07 Jul 27 19:20:38.639238 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:38.639199 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" event={"ID":"4d22b0b9-363a-4de2-aa06-6ddddf4bae90","Type":"ContainerStarted","Data":"c786f6efc4de026639410e4bedec9468a729c65e1250beb6fd42a038800d6b07"} Jul 27 19:20:40.650107 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:40.650063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" event={"ID":"4d22b0b9-363a-4de2-aa06-6ddddf4bae90","Type":"ContainerStarted","Data":"216d2a620c7506189d958572ade89aeaf1ff44b73108b3667eab0c857c99d7ca"} Jul 27 19:20:40.650475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:40.650156 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" event={"ID":"4d22b0b9-363a-4de2-aa06-6ddddf4bae90","Type":"ContainerStarted","Data":"a2a72e974feaa5f076ccb3eb2a39a326f5f5c45e24d4c50f1c9397de4db7dacb"} Jul 27 19:20:40.650475 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:40.650175 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" event={"ID":"4d22b0b9-363a-4de2-aa06-6ddddf4bae90","Type":"ContainerStarted","Data":"57bdeec69b7bc75048bee7db73304507db25911ada693807a9d4146c631488f8"} Jul 27 19:20:40.688699 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:40.688640 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-84c5c8b468-tdcsk" podStartSLOduration=1.663966817 podStartE2EDuration="3.688621101s" totalImagesPullingTime="2.024654284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:20:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:20:40.685728627 +0000 UTC m=+213.680225597" watchObservedRunningTime="2026-07-27 19:20:40.688621101 +0000 UTC m=+213.683118110" Jul 27 19:20:41.517987 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.517952 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8ffb77d6-cnphz"] Jul 27 19:20:41.521507 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.521484 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.539591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.539553 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8ffb77d6-cnphz"] Jul 27 19:20:41.625046 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625009 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-service-ca\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.625046 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625049 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-oauth-serving-cert\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.625265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625074 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-oauth-config\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.625265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625131 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-serving-cert\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.625265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625177 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-config\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.625265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-trusted-ca-bundle\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.625265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.625262 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m9qk\" (UniqueName: \"kubernetes.io/projected/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-kube-api-access-6m9qk\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726196 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726151 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-service-ca\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726196 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726190 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-oauth-serving-cert\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726212 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-oauth-config\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726229 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-serving-cert\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-config\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726409 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-trusted-ca-bundle\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.726740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.726500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m9qk\" (UniqueName: \"kubernetes.io/projected/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-kube-api-access-6m9qk\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.727174 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.727148 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-config\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.727174 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.727157 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-oauth-serving-cert\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.727294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.727232 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-service-ca\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.727420 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.727369 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-trusted-ca-bundle\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.728881 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.728853 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-oauth-config\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.729010 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.728950 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-serving-cert\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.735926 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.735889 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m9qk\" (UniqueName: \"kubernetes.io/projected/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-kube-api-access-6m9qk\") pod \"console-8ffb77d6-cnphz\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.833010 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.832917 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:20:41.983691 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:41.983655 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8ffb77d6-cnphz"] Jul 27 19:20:41.986713 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:20:41.986680 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b52bf64_6fc5_40ba_b20f_34d19f2a3497.slice/crio-cb47459ff40761949e5da11ec3860eb289f1b6fe542ba2c4975e7f1b8212d1fa WatchSource:0}: Error finding container cb47459ff40761949e5da11ec3860eb289f1b6fe542ba2c4975e7f1b8212d1fa: Status 404 returned error can't find the container with id cb47459ff40761949e5da11ec3860eb289f1b6fe542ba2c4975e7f1b8212d1fa Jul 27 19:20:42.659871 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:42.659824 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ffb77d6-cnphz" event={"ID":"3b52bf64-6fc5-40ba-b20f-34d19f2a3497","Type":"ContainerStarted","Data":"240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204"} Jul 27 19:20:42.659871 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:42.659876 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ffb77d6-cnphz" event={"ID":"3b52bf64-6fc5-40ba-b20f-34d19f2a3497","Type":"ContainerStarted","Data":"cb47459ff40761949e5da11ec3860eb289f1b6fe542ba2c4975e7f1b8212d1fa"} Jul 27 19:20:42.692211 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:42.692152 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8ffb77d6-cnphz" podStartSLOduration=1.69213043 podStartE2EDuration="1.69213043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:20:42.690878518 +0000 UTC m=+215.685375493" watchObservedRunningTime="2026-07-27 19:20:42.69213043 +0000 UTC m=+215.686627405" Jul 27 19:20:49.813039 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:49.813007 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8ffb77d6-cnphz"] Jul 27 19:20:51.833400 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:20:51.833361 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:21:00.988994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:00.988954 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fdbbdcd9-h9bvb"] Jul 27 19:21:14.833021 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:14.832976 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8ffb77d6-cnphz" podUID="3b52bf64-6fc5-40ba-b20f-34d19f2a3497" containerName="console" containerID="cri-o://240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204" gracePeriod=15 Jul 27 19:21:15.088835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.088748 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:21:15.246597 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246553 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-serving-cert\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.246597 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246594 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-service-ca\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.246878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246623 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-oauth-config\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.246878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246641 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-oauth-serving-cert\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.246878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246670 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-config\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.246878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246728 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6m9qk\" (UniqueName: \"kubernetes.io/projected/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-kube-api-access-6m9qk\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.246878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.246802 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-trusted-ca-bundle\") pod \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\" (UID: \"3b52bf64-6fc5-40ba-b20f-34d19f2a3497\") " Jul 27 19:21:15.247128 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.247047 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:15.247128 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.247095 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-service-ca" (OuterVolumeSpecName: "service-ca") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:15.247613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.247175 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-config" (OuterVolumeSpecName: "console-config") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:15.249438 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.247732 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:15.254003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.253962 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:21:15.254175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.253977 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-kube-api-access-6m9qk" (OuterVolumeSpecName: "kube-api-access-6m9qk") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "kube-api-access-6m9qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:21:15.254175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.253984 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3b52bf64-6fc5-40ba-b20f-34d19f2a3497" (UID: "3b52bf64-6fc5-40ba-b20f-34d19f2a3497"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347593 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-serving-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347627 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347639 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-oauth-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347663 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-oauth-serving-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347675 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-console-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347687 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m9qk\" (UniqueName: \"kubernetes.io/projected/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-kube-api-access-6m9qk\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.347696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.347698 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b52bf64-6fc5-40ba-b20f-34d19f2a3497-trusted-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:15.795440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.795400 2566 generic.go:356] "Generic (PLEG): container finished" podID="3b52bf64-6fc5-40ba-b20f-34d19f2a3497" containerID="240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204" exitCode=0 Jul 27 19:21:15.795603 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.795465 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8ffb77d6-cnphz" Jul 27 19:21:15.795603 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.795482 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ffb77d6-cnphz" event={"ID":"3b52bf64-6fc5-40ba-b20f-34d19f2a3497","Type":"ContainerDied","Data":"240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204"} Jul 27 19:21:15.795603 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.795540 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8ffb77d6-cnphz" event={"ID":"3b52bf64-6fc5-40ba-b20f-34d19f2a3497","Type":"ContainerDied","Data":"cb47459ff40761949e5da11ec3860eb289f1b6fe542ba2c4975e7f1b8212d1fa"} Jul 27 19:21:15.795603 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.795563 2566 scope.go:117] "RemoveContainer" containerID="240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204" Jul 27 19:21:15.803843 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.803824 2566 scope.go:117] "RemoveContainer" containerID="240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204" Jul 27 19:21:15.804250 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:21:15.804214 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204\": container with ID starting with 240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204 not found: ID does not exist" containerID="240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204" Jul 27 19:21:15.804312 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.804262 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204"} err="failed to get container status \"240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204\": rpc error: code = NotFound desc = could not find container \"240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204\": container with ID starting with 240907dbde45b8c1f23c4aafdb07cbcf0faf7d10db3e97634ce359bd4da26204 not found: ID does not exist" Jul 27 19:21:15.821044 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.821005 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8ffb77d6-cnphz"] Jul 27 19:21:15.827554 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:15.827523 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8ffb77d6-cnphz"] Jul 27 19:21:17.545472 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:17.545433 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b52bf64-6fc5-40ba-b20f-34d19f2a3497" path="/var/lib/kubelet/pods/3b52bf64-6fc5-40ba-b20f-34d19f2a3497/volumes" Jul 27 19:21:26.009610 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.009570 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-74fdbbdcd9-h9bvb" podUID="9667f9b2-188a-4de8-96ff-bb9bdbd59979" containerName="console" containerID="cri-o://ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade" gracePeriod=15 Jul 27 19:21:26.260596 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.260522 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:21:26.441409 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441372 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-oauth-serving-cert\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441409 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441427 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-config\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441643 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441464 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-service-ca\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441643 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441628 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-oauth-config\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441746 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441685 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-trusted-ca-bundle\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441837 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441808 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-552hv\" (UniqueName: \"kubernetes.io/projected/9667f9b2-188a-4de8-96ff-bb9bdbd59979-kube-api-access-552hv\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441853 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-serving-cert\") pod \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\" (UID: \"9667f9b2-188a-4de8-96ff-bb9bdbd59979\") " Jul 27 19:21:26.441943 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441895 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-service-ca" (OuterVolumeSpecName: "service-ca") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:26.441998 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.441965 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:26.442042 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.442012 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-config" (OuterVolumeSpecName: "console-config") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:26.442244 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.442219 2566 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-oauth-serving-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.442244 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.442244 2566 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.442402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.442256 2566 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.442402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.442260 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:21:26.443916 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.443887 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:21:26.444040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.443926 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:21:26.444040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.443940 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9667f9b2-188a-4de8-96ff-bb9bdbd59979-kube-api-access-552hv" (OuterVolumeSpecName: "kube-api-access-552hv") pod "9667f9b2-188a-4de8-96ff-bb9bdbd59979" (UID: "9667f9b2-188a-4de8-96ff-bb9bdbd59979"). InnerVolumeSpecName "kube-api-access-552hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:21:26.543033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.542938 2566 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-oauth-config\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.543033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.542973 2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9667f9b2-188a-4de8-96ff-bb9bdbd59979-trusted-ca-bundle\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.543033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.542983 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552hv\" (UniqueName: \"kubernetes.io/projected/9667f9b2-188a-4de8-96ff-bb9bdbd59979-kube-api-access-552hv\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.543033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.542992 2566 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9667f9b2-188a-4de8-96ff-bb9bdbd59979-console-serving-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:21:26.842965 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.842874 2566 generic.go:356] "Generic (PLEG): container finished" podID="9667f9b2-188a-4de8-96ff-bb9bdbd59979" containerID="ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade" exitCode=0 Jul 27 19:21:26.842965 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.842917 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74fdbbdcd9-h9bvb" Jul 27 19:21:26.843149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.842957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fdbbdcd9-h9bvb" event={"ID":"9667f9b2-188a-4de8-96ff-bb9bdbd59979","Type":"ContainerDied","Data":"ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade"} Jul 27 19:21:26.843149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.843008 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74fdbbdcd9-h9bvb" event={"ID":"9667f9b2-188a-4de8-96ff-bb9bdbd59979","Type":"ContainerDied","Data":"2e17f51771515c6804abafe33b40ccfc925875ff5f8e6ba8da678ad152f84830"} Jul 27 19:21:26.843149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.843028 2566 scope.go:117] "RemoveContainer" containerID="ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade" Jul 27 19:21:26.852368 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.852343 2566 scope.go:117] "RemoveContainer" containerID="ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade" Jul 27 19:21:26.852692 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:21:26.852671 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade\": container with ID starting with ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade not found: ID does not exist" containerID="ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade" Jul 27 19:21:26.852780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.852705 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade"} err="failed to get container status \"ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade\": rpc error: code = NotFound desc = could not find container \"ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade\": container with ID starting with ae333a78a3c020f4c674d71605d5c6be30755e4fa509ccf98b720bc0a7c9fade not found: ID does not exist" Jul 27 19:21:26.864664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.864626 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-74fdbbdcd9-h9bvb"] Jul 27 19:21:26.869144 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:26.869107 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-74fdbbdcd9-h9bvb"] Jul 27 19:21:27.549915 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:21:27.549880 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9667f9b2-188a-4de8-96ff-bb9bdbd59979" path="/var/lib/kubelet/pods/9667f9b2-188a-4de8-96ff-bb9bdbd59979/volumes" Jul 27 19:22:07.447149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:07.447106 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:22:07.456041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:07.456016 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:22:07.458280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:07.458259 2566 kubelet.go:1648] "Image garbage collection succeeded" Jul 27 19:22:26.509998 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.509958 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp"] Jul 27 19:22:26.512720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.510279 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9667f9b2-188a-4de8-96ff-bb9bdbd59979" containerName="console" Jul 27 19:22:26.512720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.510292 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9667f9b2-188a-4de8-96ff-bb9bdbd59979" containerName="console" Jul 27 19:22:26.512720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.510306 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b52bf64-6fc5-40ba-b20f-34d19f2a3497" containerName="console" Jul 27 19:22:26.512720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.510311 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b52bf64-6fc5-40ba-b20f-34d19f2a3497" containerName="console" Jul 27 19:22:26.512720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.510370 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b52bf64-6fc5-40ba-b20f-34d19f2a3497" containerName="console" Jul 27 19:22:26.512720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.510378 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9667f9b2-188a-4de8-96ff-bb9bdbd59979" containerName="console" Jul 27 19:22:26.519023 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.518995 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.521440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.521416 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 27 19:22:26.521965 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.521939 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp"] Jul 27 19:22:26.522298 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.522280 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 27 19:22:26.522369 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.522314 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-kz5dv\"" Jul 27 19:22:26.550775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.550724 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eeaa59f2-837a-449f-b868-79432e8f3d9e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kwrrp\" (UID: \"eeaa59f2-837a-449f-b868-79432e8f3d9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.550933 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.550801 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v244l\" (UniqueName: \"kubernetes.io/projected/eeaa59f2-837a-449f-b868-79432e8f3d9e-kube-api-access-v244l\") pod \"cert-manager-cainjector-559cdd8c87-kwrrp\" (UID: \"eeaa59f2-837a-449f-b868-79432e8f3d9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.651661 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.651623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eeaa59f2-837a-449f-b868-79432e8f3d9e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kwrrp\" (UID: \"eeaa59f2-837a-449f-b868-79432e8f3d9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.651867 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.651674 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v244l\" (UniqueName: \"kubernetes.io/projected/eeaa59f2-837a-449f-b868-79432e8f3d9e-kube-api-access-v244l\") pod \"cert-manager-cainjector-559cdd8c87-kwrrp\" (UID: \"eeaa59f2-837a-449f-b868-79432e8f3d9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.661913 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.661888 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v244l\" (UniqueName: \"kubernetes.io/projected/eeaa59f2-837a-449f-b868-79432e8f3d9e-kube-api-access-v244l\") pod \"cert-manager-cainjector-559cdd8c87-kwrrp\" (UID: \"eeaa59f2-837a-449f-b868-79432e8f3d9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.663029 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.663010 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eeaa59f2-837a-449f-b868-79432e8f3d9e-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-kwrrp\" (UID: \"eeaa59f2-837a-449f-b868-79432e8f3d9e\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.841427 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.841341 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" Jul 27 19:22:26.971821 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.971787 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp"] Jul 27 19:22:26.975051 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:22:26.975013 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeeaa59f2_837a_449f_b868_79432e8f3d9e.slice/crio-85c057e57b6df5386741474c87d5342e185661be48a0321d28c12182f345841f WatchSource:0}: Error finding container 85c057e57b6df5386741474c87d5342e185661be48a0321d28c12182f345841f: Status 404 returned error can't find the container with id 85c057e57b6df5386741474c87d5342e185661be48a0321d28c12182f345841f Jul 27 19:22:26.976952 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:26.976933 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:22:27.090075 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:27.090041 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" event={"ID":"eeaa59f2-837a-449f-b868-79432e8f3d9e","Type":"ContainerStarted","Data":"85c057e57b6df5386741474c87d5342e185661be48a0321d28c12182f345841f"} Jul 27 19:22:31.110657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:31.110619 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" event={"ID":"eeaa59f2-837a-449f-b868-79432e8f3d9e","Type":"ContainerStarted","Data":"4c29c349173291e17c56cf28583e0d4ce77856ea8cc24a7739c71f1f96de9c35"} Jul 27 19:22:34.247658 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.247600 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-kwrrp" podStartSLOduration=4.93445576 podStartE2EDuration="8.247583834s" totalImagesPullingTime="3.313128074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:22:31.127987264 +0000 UTC m=+324.122484240" watchObservedRunningTime="2026-07-27 19:22:34.247583834 +0000 UTC m=+327.242080809" Jul 27 19:22:34.248632 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.248615 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tpdfq"] Jul 27 19:22:34.251942 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.251919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.255126 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.255107 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-c78mn\"" Jul 27 19:22:34.264716 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.264688 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tpdfq"] Jul 27 19:22:34.317093 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.317057 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f9e610-06c5-4dae-b039-3e1169f2ea4a-bound-sa-token\") pod \"cert-manager-55b4d954d5-tpdfq\" (UID: \"76f9e610-06c5-4dae-b039-3e1169f2ea4a\") " pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.317269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.317122 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9g5\" (UniqueName: \"kubernetes.io/projected/76f9e610-06c5-4dae-b039-3e1169f2ea4a-kube-api-access-xx9g5\") pod \"cert-manager-55b4d954d5-tpdfq\" (UID: \"76f9e610-06c5-4dae-b039-3e1169f2ea4a\") " pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.418231 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.418188 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f9e610-06c5-4dae-b039-3e1169f2ea4a-bound-sa-token\") pod \"cert-manager-55b4d954d5-tpdfq\" (UID: \"76f9e610-06c5-4dae-b039-3e1169f2ea4a\") " pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.418411 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.418258 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9g5\" (UniqueName: \"kubernetes.io/projected/76f9e610-06c5-4dae-b039-3e1169f2ea4a-kube-api-access-xx9g5\") pod \"cert-manager-55b4d954d5-tpdfq\" (UID: \"76f9e610-06c5-4dae-b039-3e1169f2ea4a\") " pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.426652 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.426618 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/76f9e610-06c5-4dae-b039-3e1169f2ea4a-bound-sa-token\") pod \"cert-manager-55b4d954d5-tpdfq\" (UID: \"76f9e610-06c5-4dae-b039-3e1169f2ea4a\") " pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.426854 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.426831 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9g5\" (UniqueName: \"kubernetes.io/projected/76f9e610-06c5-4dae-b039-3e1169f2ea4a-kube-api-access-xx9g5\") pod \"cert-manager-55b4d954d5-tpdfq\" (UID: \"76f9e610-06c5-4dae-b039-3e1169f2ea4a\") " pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.560721 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.560617 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-tpdfq" Jul 27 19:22:34.694218 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:34.694194 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-tpdfq"] Jul 27 19:22:34.696771 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:22:34.696738 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76f9e610_06c5_4dae_b039_3e1169f2ea4a.slice/crio-efae83481822b1a223bf258c1f10205cecb0043a05a73624b075cb5ca95c4338 WatchSource:0}: Error finding container efae83481822b1a223bf258c1f10205cecb0043a05a73624b075cb5ca95c4338: Status 404 returned error can't find the container with id efae83481822b1a223bf258c1f10205cecb0043a05a73624b075cb5ca95c4338 Jul 27 19:22:35.128045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:35.128001 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tpdfq" event={"ID":"76f9e610-06c5-4dae-b039-3e1169f2ea4a","Type":"ContainerStarted","Data":"b0d4cb0e7a74e19f7dd9b2f8cd1c341e5f9314e3599284be8d5939b40b4e5dec"} Jul 27 19:22:35.128226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:35.128056 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-tpdfq" event={"ID":"76f9e610-06c5-4dae-b039-3e1169f2ea4a","Type":"ContainerStarted","Data":"efae83481822b1a223bf258c1f10205cecb0043a05a73624b075cb5ca95c4338"} Jul 27 19:22:35.146703 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:35.146649 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-tpdfq" podStartSLOduration=1.146632976 podStartE2EDuration="1.146632976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:22:35.14545625 +0000 UTC m=+328.139953226" watchObservedRunningTime="2026-07-27 19:22:35.146632976 +0000 UTC m=+328.141129950" Jul 27 19:22:57.027746 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.027707 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79"] Jul 27 19:22:57.031314 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.031293 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.037242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.037211 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-92f7h\"" Jul 27 19:22:57.037456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.037250 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 27 19:22:57.037456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.037250 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 27 19:22:57.037578 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.037456 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 27 19:22:57.037915 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.037894 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 27 19:22:57.068928 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.068895 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79"] Jul 27 19:22:57.115004 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.114966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc902e20-17ff-4d29-80fa-5acf84729993-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.115004 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.115009 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc902e20-17ff-4d29-80fa-5acf84729993-tmp\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.115223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.115040 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc902e20-17ff-4d29-80fa-5acf84729993-manifests\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.115223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.115141 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzts\" (UniqueName: \"kubernetes.io/projected/dc902e20-17ff-4d29-80fa-5acf84729993-kube-api-access-ljzts\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.115223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.115200 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc902e20-17ff-4d29-80fa-5acf84729993-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.216618 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.216571 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc902e20-17ff-4d29-80fa-5acf84729993-tmp\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.216618 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.216623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc902e20-17ff-4d29-80fa-5acf84729993-manifests\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.216874 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.216678 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzts\" (UniqueName: \"kubernetes.io/projected/dc902e20-17ff-4d29-80fa-5acf84729993-kube-api-access-ljzts\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.216874 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.216744 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc902e20-17ff-4d29-80fa-5acf84729993-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.216874 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.216817 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc902e20-17ff-4d29-80fa-5acf84729993-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.217121 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.217099 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc902e20-17ff-4d29-80fa-5acf84729993-manifests\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.219063 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.219037 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc902e20-17ff-4d29-80fa-5acf84729993-tmp\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.219293 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.219274 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc902e20-17ff-4d29-80fa-5acf84729993-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.219464 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.219449 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc902e20-17ff-4d29-80fa-5acf84729993-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.230157 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.230118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzts\" (UniqueName: \"kubernetes.io/projected/dc902e20-17ff-4d29-80fa-5acf84729993-kube-api-access-ljzts\") pod \"opendatahub-operator-controller-manager-7d79566f86-qml79\" (UID: \"dc902e20-17ff-4d29-80fa-5acf84729993\") " pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.342914 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.342829 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:22:57.483747 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:57.483708 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79"] Jul 27 19:22:57.488263 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:22:57.488237 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc902e20_17ff_4d29_80fa_5acf84729993.slice/crio-3efbb0ddc4ba94dce142d9a0be25ad23ef3a1b639114a24cff392c3375d45808 WatchSource:0}: Error finding container 3efbb0ddc4ba94dce142d9a0be25ad23ef3a1b639114a24cff392c3375d45808: Status 404 returned error can't find the container with id 3efbb0ddc4ba94dce142d9a0be25ad23ef3a1b639114a24cff392c3375d45808 Jul 27 19:22:58.237924 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:22:58.237872 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" event={"ID":"dc902e20-17ff-4d29-80fa-5acf84729993","Type":"ContainerStarted","Data":"3efbb0ddc4ba94dce142d9a0be25ad23ef3a1b639114a24cff392c3375d45808"} Jul 27 19:23:01.255440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:01.255395 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" event={"ID":"dc902e20-17ff-4d29-80fa-5acf84729993","Type":"ContainerStarted","Data":"71f68fe102631644ab34ee4e82b6483b1a17d74fdbb8fc945bbff3444fe616ff"} Jul 27 19:23:02.260850 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:02.260811 2566 generic.go:356] "Generic (PLEG): container finished" podID="dc902e20-17ff-4d29-80fa-5acf84729993" containerID="71f68fe102631644ab34ee4e82b6483b1a17d74fdbb8fc945bbff3444fe616ff" exitCode=0 Jul 27 19:23:02.261336 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:02.260891 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" event={"ID":"dc902e20-17ff-4d29-80fa-5acf84729993","Type":"ContainerDied","Data":"71f68fe102631644ab34ee4e82b6483b1a17d74fdbb8fc945bbff3444fe616ff"} Jul 27 19:23:03.267228 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:03.267193 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" event={"ID":"dc902e20-17ff-4d29-80fa-5acf84729993","Type":"ContainerStarted","Data":"ead20db997e54526cef7768fdb3beb40a1b2f3fcbf95e76d368d7d28c0536854"} Jul 27 19:23:03.267620 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:03.267292 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:23:03.289537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:03.289481 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" podStartSLOduration=1.972101367 podStartE2EDuration="6.289465178s" totalImagesPullingTime="3.293218419s" totalInitContainerRuntime="1.024145392s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:22:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:03.288326597 +0000 UTC m=+356.282823583" watchObservedRunningTime="2026-07-27 19:23:03.289465178 +0000 UTC m=+356.283962154" Jul 27 19:23:14.273843 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:14.273809 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d79566f86-qml79" Jul 27 19:23:17.318706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.318676 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-949fd4987-nqzgw"] Jul 27 19:23:17.330594 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.330565 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.337374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.337353 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 27 19:23:17.337945 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.337925 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mrm97\"" Jul 27 19:23:17.338027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.337962 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 27 19:23:17.351737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.351709 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-949fd4987-nqzgw"] Jul 27 19:23:17.392682 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.392646 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8pj8\" (UniqueName: \"kubernetes.io/projected/ddd49638-fe81-4f70-aa89-3858efdfacb2-kube-api-access-l8pj8\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.392881 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.392728 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd49638-fe81-4f70-aa89-3858efdfacb2-tls-certs\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.392881 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.392782 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddd49638-fe81-4f70-aa89-3858efdfacb2-tmp\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.441674 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.441637 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hkcrl"] Jul 27 19:23:17.445187 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.445168 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:17.456634 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.456609 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 27 19:23:17.469462 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.469433 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-2kcgs\"" Jul 27 19:23:17.477774 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.477729 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hkcrl"] Jul 27 19:23:17.493722 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.493691 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8pj8\" (UniqueName: \"kubernetes.io/projected/ddd49638-fe81-4f70-aa89-3858efdfacb2-kube-api-access-l8pj8\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.493894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.493731 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adacd43a-7371-4790-ae78-409decc5623b-cert\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:17.493894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.493772 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spnbh\" (UniqueName: \"kubernetes.io/projected/adacd43a-7371-4790-ae78-409decc5623b-kube-api-access-spnbh\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:17.493968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.493884 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd49638-fe81-4f70-aa89-3858efdfacb2-tls-certs\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.493968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.493915 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddd49638-fe81-4f70-aa89-3858efdfacb2-tmp\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.496228 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.496198 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ddd49638-fe81-4f70-aa89-3858efdfacb2-tmp\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.496518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.496497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddd49638-fe81-4f70-aa89-3858efdfacb2-tls-certs\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.524509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.524477 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8pj8\" (UniqueName: \"kubernetes.io/projected/ddd49638-fe81-4f70-aa89-3858efdfacb2-kube-api-access-l8pj8\") pod \"kube-auth-proxy-949fd4987-nqzgw\" (UID: \"ddd49638-fe81-4f70-aa89-3858efdfacb2\") " pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.594917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.594739 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adacd43a-7371-4790-ae78-409decc5623b-cert\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:17.594917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.594880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spnbh\" (UniqueName: \"kubernetes.io/projected/adacd43a-7371-4790-ae78-409decc5623b-kube-api-access-spnbh\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:17.594917 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:17.594888 2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 27 19:23:17.595190 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:17.594959 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adacd43a-7371-4790-ae78-409decc5623b-cert podName:adacd43a-7371-4790-ae78-409decc5623b nodeName:}" failed. No retries permitted until 2026-07-27 19:23:18.094938884 +0000 UTC m=+371.089435837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/adacd43a-7371-4790-ae78-409decc5623b-cert") pod "odh-model-controller-7998c67dc4-hkcrl" (UID: "adacd43a-7371-4790-ae78-409decc5623b") : secret "odh-model-controller-webhook-cert" not found Jul 27 19:23:17.610075 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.610043 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spnbh\" (UniqueName: \"kubernetes.io/projected/adacd43a-7371-4790-ae78-409decc5623b-kube-api-access-spnbh\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:17.640029 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.639993 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" Jul 27 19:23:17.792925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:17.792710 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-949fd4987-nqzgw"] Jul 27 19:23:17.796361 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:17.796322 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd49638_fe81_4f70_aa89_3858efdfacb2.slice/crio-18ab231af424bf00dfe9b2c1695de59fb5053f9181c2bf7c1844797233a725d7 WatchSource:0}: Error finding container 18ab231af424bf00dfe9b2c1695de59fb5053f9181c2bf7c1844797233a725d7: Status 404 returned error can't find the container with id 18ab231af424bf00dfe9b2c1695de59fb5053f9181c2bf7c1844797233a725d7 Jul 27 19:23:18.098218 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.098174 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adacd43a-7371-4790-ae78-409decc5623b-cert\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:18.100562 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.100541 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/adacd43a-7371-4790-ae78-409decc5623b-cert\") pod \"odh-model-controller-7998c67dc4-hkcrl\" (UID: \"adacd43a-7371-4790-ae78-409decc5623b\") " pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:18.333058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.333023 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" event={"ID":"ddd49638-fe81-4f70-aa89-3858efdfacb2","Type":"ContainerStarted","Data":"18ab231af424bf00dfe9b2c1695de59fb5053f9181c2bf7c1844797233a725d7"} Jul 27 19:23:18.355362 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.355276 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:18.500333 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.500296 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r7wzv"] Jul 27 19:23:18.504161 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.504135 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:18.506902 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.506873 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 27 19:23:18.507053 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.506924 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-fr6gm\"" Jul 27 19:23:18.514294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.514264 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r7wzv"] Jul 27 19:23:18.521708 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.521328 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-hkcrl"] Jul 27 19:23:18.526459 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:18.526421 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadacd43a_7371_4790_ae78_409decc5623b.slice/crio-66c3f4ba9c1c42190d11dc929dff90ea61e31400afa0cf8123b217d5b33be833 WatchSource:0}: Error finding container 66c3f4ba9c1c42190d11dc929dff90ea61e31400afa0cf8123b217d5b33be833: Status 404 returned error can't find the container with id 66c3f4ba9c1c42190d11dc929dff90ea61e31400afa0cf8123b217d5b33be833 Jul 27 19:23:18.602507 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.602463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa666236-ec46-48f4-af3c-32ce4e76e385-tls-certs\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:18.602845 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.602795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bctcd\" (UniqueName: \"kubernetes.io/projected/fa666236-ec46-48f4-af3c-32ce4e76e385-kube-api-access-bctcd\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:18.703866 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.703833 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa666236-ec46-48f4-af3c-32ce4e76e385-tls-certs\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:18.703866 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.703880 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bctcd\" (UniqueName: \"kubernetes.io/projected/fa666236-ec46-48f4-af3c-32ce4e76e385-kube-api-access-bctcd\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:18.704127 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:18.704011 2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 27 19:23:18.704127 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:18.704095 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa666236-ec46-48f4-af3c-32ce4e76e385-tls-certs podName:fa666236-ec46-48f4-af3c-32ce4e76e385 nodeName:}" failed. No retries permitted until 2026-07-27 19:23:19.204073663 +0000 UTC m=+372.198570623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fa666236-ec46-48f4-af3c-32ce4e76e385-tls-certs") pod "model-serving-api-54d89bd5fc-r7wzv" (UID: "fa666236-ec46-48f4-af3c-32ce4e76e385") : secret "model-serving-api-tls" not found Jul 27 19:23:18.717659 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.717628 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bctcd\" (UniqueName: \"kubernetes.io/projected/fa666236-ec46-48f4-af3c-32ce4e76e385-kube-api-access-bctcd\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:18.722262 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.722226 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4"] Jul 27 19:23:18.726795 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.726600 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.729445 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.729422 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 27 19:23:18.729571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.729426 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 27 19:23:18.729925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.729905 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 27 19:23:18.730457 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.730435 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 27 19:23:18.730998 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.730976 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sskzp\"" Jul 27 19:23:18.750399 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.750286 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4"] Jul 27 19:23:18.805323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805280 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.805512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805338 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.805512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805395 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slcj6\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-kube-api-access-slcj6\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.805512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.805667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805532 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.805667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805563 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.805667 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.805637 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd01d6f-8d75-4669-876c-81bae959d29f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906235 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906197 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906235 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906247 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906271 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slcj6\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-kube-api-access-slcj6\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906326 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906357 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906436 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.906678 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.906487 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd01d6f-8d75-4669-876c-81bae959d29f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.907116 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.907078 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.909172 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.909144 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd01d6f-8d75-4669-876c-81bae959d29f-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.909433 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.909413 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.909525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.909452 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.909578 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.909556 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.932298 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.932267 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slcj6\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-kube-api-access-slcj6\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:18.933242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:18.933219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-bp8j4\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:19.039396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.039304 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:19.210106 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.209505 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa666236-ec46-48f4-af3c-32ce4e76e385-tls-certs\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:19.211969 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.211940 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4"] Jul 27 19:23:19.213256 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.213221 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa666236-ec46-48f4-af3c-32ce4e76e385-tls-certs\") pod \"model-serving-api-54d89bd5fc-r7wzv\" (UID: \"fa666236-ec46-48f4-af3c-32ce4e76e385\") " pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:19.213542 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:19.213512 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd01d6f_8d75_4669_876c_81bae959d29f.slice/crio-8c55bfe109ff1e53697d0def1730469fdb3745bc5b654f5c2ba88404471a557c WatchSource:0}: Error finding container 8c55bfe109ff1e53697d0def1730469fdb3745bc5b654f5c2ba88404471a557c: Status 404 returned error can't find the container with id 8c55bfe109ff1e53697d0def1730469fdb3745bc5b654f5c2ba88404471a557c Jul 27 19:23:19.343025 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.342926 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" event={"ID":"adacd43a-7371-4790-ae78-409decc5623b","Type":"ContainerStarted","Data":"66c3f4ba9c1c42190d11dc929dff90ea61e31400afa0cf8123b217d5b33be833"} Jul 27 19:23:19.344815 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.344778 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" event={"ID":"5fd01d6f-8d75-4669-876c-81bae959d29f","Type":"ContainerStarted","Data":"8c55bfe109ff1e53697d0def1730469fdb3745bc5b654f5c2ba88404471a557c"} Jul 27 19:23:19.427674 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.427521 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:19.615223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:19.615180 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-r7wzv"] Jul 27 19:23:20.369272 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:20.369238 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa666236_ec46_48f4_af3c_32ce4e76e385.slice/crio-d6d2be2cd0ac8e4ea93667ccd3ad01e03ec8fca4bbc2a1d19b00cbe16fb34914 WatchSource:0}: Error finding container d6d2be2cd0ac8e4ea93667ccd3ad01e03ec8fca4bbc2a1d19b00cbe16fb34914: Status 404 returned error can't find the container with id d6d2be2cd0ac8e4ea93667ccd3ad01e03ec8fca4bbc2a1d19b00cbe16fb34914 Jul 27 19:23:21.359209 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:21.359165 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" event={"ID":"fa666236-ec46-48f4-af3c-32ce4e76e385","Type":"ContainerStarted","Data":"d6d2be2cd0ac8e4ea93667ccd3ad01e03ec8fca4bbc2a1d19b00cbe16fb34914"} Jul 27 19:23:22.369832 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:22.369791 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" event={"ID":"ddd49638-fe81-4f70-aa89-3858efdfacb2","Type":"ContainerStarted","Data":"efb46fff2d2bc3212e9366c4780ed2503bd4a280528ad6a0028ae73654258d0e"} Jul 27 19:23:22.371356 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:22.371327 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" event={"ID":"adacd43a-7371-4790-ae78-409decc5623b","Type":"ContainerStarted","Data":"cde20fd938c397fae3b4366415e9271505ef38a3d10ec2b8a32db3cfc44d5707"} Jul 27 19:23:22.371495 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:22.371426 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:22.386479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:22.386422 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-949fd4987-nqzgw" podStartSLOduration=1.398441198 podStartE2EDuration="5.386403216s" totalImagesPullingTime="3.987962018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:22.385616425 +0000 UTC m=+375.380113402" watchObservedRunningTime="2026-07-27 19:23:22.386403216 +0000 UTC m=+375.380900190" Jul 27 19:23:22.403717 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:22.403166 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" podStartSLOduration=2.098522369 podStartE2EDuration="5.403141247s" totalImagesPullingTime="3.304618878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:22.402113722 +0000 UTC m=+375.396610697" watchObservedRunningTime="2026-07-27 19:23:22.403141247 +0000 UTC m=+375.397638224" Jul 27 19:23:23.683892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:23.683850 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:23:23.684229 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:23.683921 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:23:24.385583 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.385537 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" event={"ID":"5fd01d6f-8d75-4669-876c-81bae959d29f","Type":"ContainerStarted","Data":"2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8"} Jul 27 19:23:24.386740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.386710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:24.388810 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.388780 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" event={"ID":"fa666236-ec46-48f4-af3c-32ce4e76e385","Type":"ContainerStarted","Data":"6d383def96af5311078aa76041c3001478205dffc319ba61cd513969e444141d"} Jul 27 19:23:24.388941 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.388797 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bp8j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:23:24.389023 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.388930 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:23:24.389084 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.389030 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:23:24.434098 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.434030 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" podStartSLOduration=1.9669639380000001 podStartE2EDuration="6.434005372s" totalImagesPullingTime="4.467041434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:24.431843457 +0000 UTC m=+377.426340428" watchObservedRunningTime="2026-07-27 19:23:24.434005372 +0000 UTC m=+377.428502351" Jul 27 19:23:24.453834 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:24.453777 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" podStartSLOduration=3.1410077 podStartE2EDuration="6.453743853s" totalImagesPullingTime="3.312736153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:24.453337713 +0000 UTC m=+377.447834691" watchObservedRunningTime="2026-07-27 19:23:24.453743853 +0000 UTC m=+377.448240829" Jul 27 19:23:25.396494 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.396451 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bp8j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:23:25.397015 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.396515 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:23:25.684961 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.684922 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q"] Jul 27 19:23:25.688608 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.688588 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.691611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.691585 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-x2f7w\"" Jul 27 19:23:25.716614 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.716579 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q"] Jul 27 19:23:25.780497 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780429 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3437a2-8940-4738-8105-dd572305c90e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.780497 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.780811 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780541 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.780811 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780691 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhrzt\" (UniqueName: \"kubernetes.io/projected/4d3437a2-8940-4738-8105-dd572305c90e-kube-api-access-bhrzt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.780811 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780728 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.780811 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780795 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.781019 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780831 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3437a2-8940-4738-8105-dd572305c90e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.781019 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780873 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.781019 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.780939 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3437a2-8940-4738-8105-dd572305c90e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882104 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhrzt\" (UniqueName: \"kubernetes.io/projected/4d3437a2-8940-4738-8105-dd572305c90e-kube-api-access-bhrzt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882144 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882285 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3437a2-8940-4738-8105-dd572305c90e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882330 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882375 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3437a2-8940-4738-8105-dd572305c90e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882420 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3437a2-8940-4738-8105-dd572305c90e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882468 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882495 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882883 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882832 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.882937 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.882901 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.883323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.883272 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.884380 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.884330 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d3437a2-8940-4738-8105-dd572305c90e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.885614 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.885590 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d3437a2-8940-4738-8105-dd572305c90e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.885712 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.885591 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d3437a2-8940-4738-8105-dd572305c90e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.891716 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.891687 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d3437a2-8940-4738-8105-dd572305c90e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:25.891927 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:25.891903 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhrzt\" (UniqueName: \"kubernetes.io/projected/4d3437a2-8940-4738-8105-dd572305c90e-kube-api-access-bhrzt\") pod \"data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q\" (UID: \"4d3437a2-8940-4738-8105-dd572305c90e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:26.000293 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:26.000188 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:26.173181 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:26.173145 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q"] Jul 27 19:23:26.174412 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:26.174375 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3437a2_8940_4738_8105_dd572305c90e.slice/crio-05648094b74af26337a397e19a226fac2b16e4be807839192a57c6c706e22f9a WatchSource:0}: Error finding container 05648094b74af26337a397e19a226fac2b16e4be807839192a57c6c706e22f9a: Status 404 returned error can't find the container with id 05648094b74af26337a397e19a226fac2b16e4be807839192a57c6c706e22f9a Jul 27 19:23:26.401840 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:26.401738 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" event={"ID":"4d3437a2-8940-4738-8105-dd572305c90e","Type":"ContainerStarted","Data":"05648094b74af26337a397e19a226fac2b16e4be807839192a57c6c706e22f9a"} Jul 27 19:23:26.402561 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:26.402540 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:28.093704 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.093670 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg"] Jul 27 19:23:28.097809 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.097755 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.101084 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.101058 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-b2bv4\"" Jul 27 19:23:28.101217 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.101154 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 27 19:23:28.107657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.107608 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf"] Jul 27 19:23:28.114936 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.114895 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg"] Jul 27 19:23:28.115114 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.115028 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.117569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.117537 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 27 19:23:28.117722 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.117699 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w9dj8\"" Jul 27 19:23:28.129149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.128964 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf"] Jul 27 19:23:28.205894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.205853 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63021b5-abd9-47ce-95f8-540a45408c39-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.205894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.205903 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248dea2e-c215-43ba-b4a7-bde0c0581a2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.206127 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.206060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hktkx\" (UniqueName: \"kubernetes.io/projected/f63021b5-abd9-47ce-95f8-540a45408c39-kube-api-access-hktkx\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.206127 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.206100 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtwk2\" (UniqueName: \"kubernetes.io/projected/248dea2e-c215-43ba-b4a7-bde0c0581a2d-kube-api-access-xtwk2\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.306701 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.306662 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hktkx\" (UniqueName: \"kubernetes.io/projected/f63021b5-abd9-47ce-95f8-540a45408c39-kube-api-access-hktkx\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.306884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.306711 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtwk2\" (UniqueName: \"kubernetes.io/projected/248dea2e-c215-43ba-b4a7-bde0c0581a2d-kube-api-access-xtwk2\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.306884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.306748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63021b5-abd9-47ce-95f8-540a45408c39-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.306884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.306791 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248dea2e-c215-43ba-b4a7-bde0c0581a2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.306884 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:28.306874 2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 27 19:23:28.307107 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:28.306942 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f63021b5-abd9-47ce-95f8-540a45408c39-cert podName:f63021b5-abd9-47ce-95f8-540a45408c39 nodeName:}" failed. No retries permitted until 2026-07-27 19:23:28.80692335 +0000 UTC m=+381.801420308 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f63021b5-abd9-47ce-95f8-540a45408c39-cert") pod "llmisvc-controller-manager-777c6bd7d5-bj7cf" (UID: "f63021b5-abd9-47ce-95f8-540a45408c39") : secret "llmisvc-webhook-server-cert" not found Jul 27 19:23:28.307107 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:28.306966 2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found Jul 27 19:23:28.307107 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:28.307020 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/248dea2e-c215-43ba-b4a7-bde0c0581a2d-cert podName:248dea2e-c215-43ba-b4a7-bde0c0581a2d nodeName:}" failed. No retries permitted until 2026-07-27 19:23:28.807001411 +0000 UTC m=+381.801498367 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/248dea2e-c215-43ba-b4a7-bde0c0581a2d-cert") pod "kserve-controller-manager-d6f4b65b8-kb4qg" (UID: "248dea2e-c215-43ba-b4a7-bde0c0581a2d") : secret "kserve-webhook-server-cert" not found Jul 27 19:23:28.317509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.317479 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtwk2\" (UniqueName: \"kubernetes.io/projected/248dea2e-c215-43ba-b4a7-bde0c0581a2d-kube-api-access-xtwk2\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.317890 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.317871 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hktkx\" (UniqueName: \"kubernetes.io/projected/f63021b5-abd9-47ce-95f8-540a45408c39-kube-api-access-hktkx\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.813118 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.813071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63021b5-abd9-47ce-95f8-540a45408c39-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.813118 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.813116 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248dea2e-c215-43ba-b4a7-bde0c0581a2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:28.816105 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.816080 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f63021b5-abd9-47ce-95f8-540a45408c39-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bj7cf\" (UID: \"f63021b5-abd9-47ce-95f8-540a45408c39\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:28.816243 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:28.816084 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/248dea2e-c215-43ba-b4a7-bde0c0581a2d-cert\") pod \"kserve-controller-manager-d6f4b65b8-kb4qg\" (UID: \"248dea2e-c215-43ba-b4a7-bde0c0581a2d\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:29.013595 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.013557 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:29.030430 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.030389 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:29.314706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.314545 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:23:29.314706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.314643 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:23:29.315355 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.314719 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:23:29.380962 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.380472 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4"] Jul 27 19:23:29.380962 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.380821 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" containerID="cri-o://2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8" gracePeriod=30 Jul 27 19:23:29.403248 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.403213 2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-bp8j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:8080/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" start-of-body= Jul 27 19:23:29.403410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.403282 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.32:8080/ready\": dial tcp 10.132.0.32:8080: connect: connection refused" Jul 27 19:23:29.416227 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.416187 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg"] Jul 27 19:23:29.419222 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:29.419188 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod248dea2e_c215_43ba_b4a7_bde0c0581a2d.slice/crio-c58d29a52bc4c1d5749efba96b0b381f1277fabae81f8036e1d0383b7458f8e2 WatchSource:0}: Error finding container c58d29a52bc4c1d5749efba96b0b381f1277fabae81f8036e1d0383b7458f8e2: Status 404 returned error can't find the container with id c58d29a52bc4c1d5749efba96b0b381f1277fabae81f8036e1d0383b7458f8e2 Jul 27 19:23:29.421816 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.421664 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" event={"ID":"4d3437a2-8940-4738-8105-dd572305c90e","Type":"ContainerStarted","Data":"e6ad31c244fca5c7021db12b67afbd19c7a9a61e1128c97cbae7705938cb998c"} Jul 27 19:23:29.443775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.443726 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf"] Jul 27 19:23:29.464827 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:23:29.464786 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf63021b5_abd9_47ce_95f8_540a45408c39.slice/crio-50c63e45389d23117af44edd8e7c5b10f0ad0f55633f5ba72b378fc59fff690e WatchSource:0}: Error finding container 50c63e45389d23117af44edd8e7c5b10f0ad0f55633f5ba72b378fc59fff690e: Status 404 returned error can't find the container with id 50c63e45389d23117af44edd8e7c5b10f0ad0f55633f5ba72b378fc59fff690e Jul 27 19:23:29.468569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.468514 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" podStartSLOduration=1.331022328 podStartE2EDuration="4.468497862s" totalImagesPullingTime="3.137475534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:29.46554923 +0000 UTC m=+382.460046204" watchObservedRunningTime="2026-07-27 19:23:29.468497862 +0000 UTC m=+382.462994837" Jul 27 19:23:29.653003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.652968 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:29.723230 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.723198 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-ca-configmap\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.723230 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.723232 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-dns-cert\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.723491 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.723254 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-cacerts\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.723709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.723684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slcj6\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-kube-api-access-slcj6\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.723878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.723685 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:23:29.723968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.723872 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-token\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.724121 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.724098 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-kubeconfig\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.724237 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.724155 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd01d6f-8d75-4669-876c-81bae959d29f-local-certs\") pod \"5fd01d6f-8d75-4669-876c-81bae959d29f\" (UID: \"5fd01d6f-8d75-4669-876c-81bae959d29f\") " Jul 27 19:23:29.724532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.724512 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-ca-configmap\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:29.726208 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.726180 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:23:29.726532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.726472 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-cacerts" (OuterVolumeSpecName: "cacerts") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:23:29.726817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.726792 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:23:29.726954 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.726936 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-kube-api-access-slcj6" (OuterVolumeSpecName: "kube-api-access-slcj6") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "kube-api-access-slcj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:23:29.727100 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.727073 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fd01d6f-8d75-4669-876c-81bae959d29f-local-certs" (OuterVolumeSpecName: "local-certs") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:23:29.727429 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.727409 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-token" (OuterVolumeSpecName: "istio-token") pod "5fd01d6f-8d75-4669-876c-81bae959d29f" (UID: "5fd01d6f-8d75-4669-876c-81bae959d29f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:23:29.825531 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.825437 2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fd01d6f-8d75-4669-876c-81bae959d29f-local-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:29.825531 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.825471 2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-csr-dns-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:29.825531 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.825482 2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-cacerts\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:29.825531 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.825491 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slcj6\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-kube-api-access-slcj6\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:29.825531 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.825501 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:29.825531 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:29.825512 2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5fd01d6f-8d75-4669-876c-81bae959d29f-istio-kubeconfig\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:23:30.001112 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.001067 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:30.007565 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.007520 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:30.428606 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.428572 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" event={"ID":"f63021b5-abd9-47ce-95f8-540a45408c39","Type":"ContainerStarted","Data":"50c63e45389d23117af44edd8e7c5b10f0ad0f55633f5ba72b378fc59fff690e"} Jul 27 19:23:30.429996 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.429966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" event={"ID":"248dea2e-c215-43ba-b4a7-bde0c0581a2d","Type":"ContainerStarted","Data":"c58d29a52bc4c1d5749efba96b0b381f1277fabae81f8036e1d0383b7458f8e2"} Jul 27 19:23:30.431412 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.431385 2566 generic.go:356] "Generic (PLEG): container finished" podID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerID="2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8" exitCode=0 Jul 27 19:23:30.431527 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.431440 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" Jul 27 19:23:30.431527 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.431476 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" event={"ID":"5fd01d6f-8d75-4669-876c-81bae959d29f","Type":"ContainerDied","Data":"2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8"} Jul 27 19:23:30.431631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.431529 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4" event={"ID":"5fd01d6f-8d75-4669-876c-81bae959d29f","Type":"ContainerDied","Data":"8c55bfe109ff1e53697d0def1730469fdb3745bc5b654f5c2ba88404471a557c"} Jul 27 19:23:30.431631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.431554 2566 scope.go:117] "RemoveContainer" containerID="2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8" Jul 27 19:23:30.432146 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.431877 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:30.433267 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.432896 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q" Jul 27 19:23:30.487491 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.487410 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4"] Jul 27 19:23:30.495366 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.495334 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-bp8j4"] Jul 27 19:23:30.647602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.647575 2566 scope.go:117] "RemoveContainer" containerID="2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8" Jul 27 19:23:30.648050 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:23:30.648017 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8\": container with ID starting with 2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8 not found: ID does not exist" containerID="2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8" Jul 27 19:23:30.648176 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:30.648062 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8"} err="failed to get container status \"2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8\": rpc error: code = NotFound desc = could not find container \"2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8\": container with ID starting with 2afd6954f4e49415510be3fd6f36c8ad60bf3884c7c94dadc26357e488db25c8 not found: ID does not exist" Jul 27 19:23:31.546320 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:31.546288 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" path="/var/lib/kubelet/pods/5fd01d6f-8d75-4669-876c-81bae959d29f/volumes" Jul 27 19:23:33.379375 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.379341 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-hkcrl" Jul 27 19:23:33.451956 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.451920 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" event={"ID":"f63021b5-abd9-47ce-95f8-540a45408c39","Type":"ContainerStarted","Data":"05995b58c401e2a4bee9cf05c5b8ca679c603721ca3f0a64ff4b189a78b3cca4"} Jul 27 19:23:33.452154 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.451988 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:23:33.453584 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.453555 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" event={"ID":"248dea2e-c215-43ba-b4a7-bde0c0581a2d","Type":"ContainerStarted","Data":"20fd07e3cd98ababe7fe8228d0b7e722a6ecb74d7237e924811b1c4612fee747"} Jul 27 19:23:33.453720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.453701 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:23:33.471487 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.471428 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" podStartSLOduration=2.172978076 podStartE2EDuration="5.471410424s" totalImagesPullingTime="3.298432348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:33.469455328 +0000 UTC m=+386.463952303" watchObservedRunningTime="2026-07-27 19:23:33.471410424 +0000 UTC m=+386.465907437" Jul 27 19:23:33.487746 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:33.487673 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" podStartSLOduration=2.126808123 podStartE2EDuration="5.487647434s" totalImagesPullingTime="3.360839311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:23:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:23:33.487108986 +0000 UTC m=+386.481605970" watchObservedRunningTime="2026-07-27 19:23:33.487647434 +0000 UTC m=+386.482144409" Jul 27 19:23:35.400866 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:23:35.400832 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-r7wzv" Jul 27 19:24:04.460450 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:04.460413 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bj7cf" Jul 27 19:24:04.463628 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:04.463607 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-kb4qg" Jul 27 19:24:17.744014 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.743971 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-56xbx"] Jul 27 19:24:17.744509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.744421 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" Jul 27 19:24:17.744509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.744434 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" Jul 27 19:24:17.744612 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.744521 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fd01d6f-8d75-4669-876c-81bae959d29f" containerName="discovery" Jul 27 19:24:17.755728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.755696 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:17.756879 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.756851 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-56xbx"] Jul 27 19:24:17.758236 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.758207 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 27 19:24:17.758392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.758214 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-n65hv\"" Jul 27 19:24:17.759093 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.759072 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 27 19:24:17.869047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.869002 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttdm\" (UniqueName: \"kubernetes.io/projected/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab-kube-api-access-5ttdm\") pod \"kuadrant-operator-catalog-56xbx\" (UID: \"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab\") " pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:17.970216 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.970173 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttdm\" (UniqueName: \"kubernetes.io/projected/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab-kube-api-access-5ttdm\") pod \"kuadrant-operator-catalog-56xbx\" (UID: \"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab\") " pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:17.978839 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:17.978804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttdm\" (UniqueName: \"kubernetes.io/projected/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab-kube-api-access-5ttdm\") pod \"kuadrant-operator-catalog-56xbx\" (UID: \"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab\") " pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:18.067876 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.067746 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:18.110062 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.110014 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-56xbx"] Jul 27 19:24:18.214601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.214566 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-56xbx"] Jul 27 19:24:18.218426 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:24:18.218396 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b8ec26d_323c_4e4b_8aee_d7daf527c1ab.slice/crio-60a021340e43267b9c49b47c6d54f2b7511bfc9e9d744c927cf888bb73ce646e WatchSource:0}: Error finding container 60a021340e43267b9c49b47c6d54f2b7511bfc9e9d744c927cf888bb73ce646e: Status 404 returned error can't find the container with id 60a021340e43267b9c49b47c6d54f2b7511bfc9e9d744c927cf888bb73ce646e Jul 27 19:24:18.321502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.321414 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rm2np"] Jul 27 19:24:18.326438 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.326414 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:18.332456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.332428 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rm2np"] Jul 27 19:24:18.373435 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.373396 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxcfs\" (UniqueName: \"kubernetes.io/projected/d46166c0-1a68-45b5-ad0c-b622351828a8-kube-api-access-sxcfs\") pod \"kuadrant-operator-catalog-rm2np\" (UID: \"d46166c0-1a68-45b5-ad0c-b622351828a8\") " pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:18.474423 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.474383 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxcfs\" (UniqueName: \"kubernetes.io/projected/d46166c0-1a68-45b5-ad0c-b622351828a8-kube-api-access-sxcfs\") pod \"kuadrant-operator-catalog-rm2np\" (UID: \"d46166c0-1a68-45b5-ad0c-b622351828a8\") " pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:18.482556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.482527 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxcfs\" (UniqueName: \"kubernetes.io/projected/d46166c0-1a68-45b5-ad0c-b622351828a8-kube-api-access-sxcfs\") pod \"kuadrant-operator-catalog-rm2np\" (UID: \"d46166c0-1a68-45b5-ad0c-b622351828a8\") " pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:18.638804 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.638696 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:18.678990 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.678954 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" event={"ID":"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab","Type":"ContainerStarted","Data":"60a021340e43267b9c49b47c6d54f2b7511bfc9e9d744c927cf888bb73ce646e"} Jul 27 19:24:18.885923 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:18.885823 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-rm2np"] Jul 27 19:24:18.932277 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:24:18.932244 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46166c0_1a68_45b5_ad0c_b622351828a8.slice/crio-579207f944b8159923fbe2265690fb565ec89b75f84485bbdc94493720654e05 WatchSource:0}: Error finding container 579207f944b8159923fbe2265690fb565ec89b75f84485bbdc94493720654e05: Status 404 returned error can't find the container with id 579207f944b8159923fbe2265690fb565ec89b75f84485bbdc94493720654e05 Jul 27 19:24:19.686258 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:19.686217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" event={"ID":"d46166c0-1a68-45b5-ad0c-b622351828a8","Type":"ContainerStarted","Data":"579207f944b8159923fbe2265690fb565ec89b75f84485bbdc94493720654e05"} Jul 27 19:24:20.693425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:20.693389 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" event={"ID":"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab","Type":"ContainerStarted","Data":"4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3"} Jul 27 19:24:20.693935 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:20.693422 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" podUID="2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" containerName="registry-server" containerID="cri-o://4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3" gracePeriod=2 Jul 27 19:24:20.695096 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:20.695067 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" event={"ID":"d46166c0-1a68-45b5-ad0c-b622351828a8","Type":"ContainerStarted","Data":"9dc906cc3d924ac32ebb4c13586e0cacf97bb99d67531399924b2e0f01d112cb"} Jul 27 19:24:20.709839 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:20.709788 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" podStartSLOduration=1.759724777 podStartE2EDuration="3.709751408s" totalImagesPullingTime="1.950026631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:20.70822448 +0000 UTC m=+433.702721449" watchObservedRunningTime="2026-07-27 19:24:20.709751408 +0000 UTC m=+433.704248533" Jul 27 19:24:20.724409 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:20.724354 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" podStartSLOduration=1.4874392300000001 podStartE2EDuration="2.724333198s" totalImagesPullingTime="1.236893968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:24:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:24:20.722925086 +0000 UTC m=+433.717422069" watchObservedRunningTime="2026-07-27 19:24:20.724333198 +0000 UTC m=+433.718830201" Jul 27 19:24:20.951723 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:20.951638 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:21.097987 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.097953 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ttdm\" (UniqueName: \"kubernetes.io/projected/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab-kube-api-access-5ttdm\") pod \"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab\" (UID: \"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab\") " Jul 27 19:24:21.100243 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.100211 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab-kube-api-access-5ttdm" (OuterVolumeSpecName: "kube-api-access-5ttdm") pod "2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" (UID: "2b8ec26d-323c-4e4b-8aee-d7daf527c1ab"). InnerVolumeSpecName "kube-api-access-5ttdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:24:21.199037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.199002 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ttdm\" (UniqueName: \"kubernetes.io/projected/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab-kube-api-access-5ttdm\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:24:21.701713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.701669 2566 generic.go:356] "Generic (PLEG): container finished" podID="2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" containerID="4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3" exitCode=0 Jul 27 19:24:21.702198 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.701727 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" Jul 27 19:24:21.702198 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.701769 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" event={"ID":"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab","Type":"ContainerDied","Data":"4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3"} Jul 27 19:24:21.702198 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.701823 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-56xbx" event={"ID":"2b8ec26d-323c-4e4b-8aee-d7daf527c1ab","Type":"ContainerDied","Data":"60a021340e43267b9c49b47c6d54f2b7511bfc9e9d744c927cf888bb73ce646e"} Jul 27 19:24:21.702198 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.701846 2566 scope.go:117] "RemoveContainer" containerID="4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3" Jul 27 19:24:21.711324 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.711298 2566 scope.go:117] "RemoveContainer" containerID="4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3" Jul 27 19:24:21.711634 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:24:21.711616 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3\": container with ID starting with 4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3 not found: ID does not exist" containerID="4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3" Jul 27 19:24:21.711678 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.711644 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3"} err="failed to get container status \"4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3\": rpc error: code = NotFound desc = could not find container \"4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3\": container with ID starting with 4ef1941d4e37fd7e90857eddb85655fcebae768bfa22fa5cf7be54498aee99a3 not found: ID does not exist" Jul 27 19:24:21.719933 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.719896 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-56xbx"] Jul 27 19:24:21.726221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:21.726182 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-56xbx"] Jul 27 19:24:23.545878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:23.545842 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" path="/var/lib/kubelet/pods/2b8ec26d-323c-4e4b-8aee-d7daf527c1ab/volumes" Jul 27 19:24:28.639124 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:28.639081 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:28.639644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:28.639150 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:28.661647 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:28.661621 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:24:28.763808 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:24:28.763778 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-rm2np" Jul 27 19:25:12.005507 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.005467 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r"] Jul 27 19:25:12.006080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.006060 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" containerName="registry-server" Jul 27 19:25:12.006156 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.006082 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" containerName="registry-server" Jul 27 19:25:12.006205 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.006188 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b8ec26d-323c-4e4b-8aee-d7daf527c1ab" containerName="registry-server" Jul 27 19:25:12.010908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.010880 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:12.014410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.014041 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 27 19:25:12.014410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.014270 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-j65lg\"" Jul 27 19:25:12.018582 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.018547 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r"] Jul 27 19:25:12.052034 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.051999 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2j6p\" (UniqueName: \"kubernetes.io/projected/f41b4a46-95f2-4430-a050-1742e1f38180-kube-api-access-v2j6p\") pod \"dns-operator-controller-manager-864dd78c4c-5wn9r\" (UID: \"f41b4a46-95f2-4430-a050-1742e1f38180\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:12.153032 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.153000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2j6p\" (UniqueName: \"kubernetes.io/projected/f41b4a46-95f2-4430-a050-1742e1f38180-kube-api-access-v2j6p\") pod \"dns-operator-controller-manager-864dd78c4c-5wn9r\" (UID: \"f41b4a46-95f2-4430-a050-1742e1f38180\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:12.169784 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.169719 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2j6p\" (UniqueName: \"kubernetes.io/projected/f41b4a46-95f2-4430-a050-1742e1f38180-kube-api-access-v2j6p\") pod \"dns-operator-controller-manager-864dd78c4c-5wn9r\" (UID: \"f41b4a46-95f2-4430-a050-1742e1f38180\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:12.325824 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.325697 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:12.475281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.475250 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r"] Jul 27 19:25:12.478155 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:12.478119 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf41b4a46_95f2_4430_a050_1742e1f38180.slice/crio-8a5b1a1aa648bc9894c1c78b243a32ae9fe2dcfa23fb34edcf2d13657cb89b49 WatchSource:0}: Error finding container 8a5b1a1aa648bc9894c1c78b243a32ae9fe2dcfa23fb34edcf2d13657cb89b49: Status 404 returned error can't find the container with id 8a5b1a1aa648bc9894c1c78b243a32ae9fe2dcfa23fb34edcf2d13657cb89b49 Jul 27 19:25:12.954962 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:12.954923 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" event={"ID":"f41b4a46-95f2-4430-a050-1742e1f38180","Type":"ContainerStarted","Data":"8a5b1a1aa648bc9894c1c78b243a32ae9fe2dcfa23fb34edcf2d13657cb89b49"} Jul 27 19:25:14.971115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:14.971077 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" event={"ID":"f41b4a46-95f2-4430-a050-1742e1f38180","Type":"ContainerStarted","Data":"b0fe836adf03e5f0315f00e5ddf988e2966423931fab3faba245fa54a6cfca45"} Jul 27 19:25:14.971591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:14.971298 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:14.991730 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:14.991678 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" podStartSLOduration=2.217180038 podStartE2EDuration="3.991658578s" totalImagesPullingTime="1.77447854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:14.98992199 +0000 UTC m=+487.984418970" watchObservedRunningTime="2026-07-27 19:25:14.991658578 +0000 UTC m=+487.986155630" Jul 27 19:25:15.500053 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.500010 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55"] Jul 27 19:25:15.503715 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.503697 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:15.506607 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.506580 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-tlnv2\"" Jul 27 19:25:15.530104 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.530067 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55"] Jul 27 19:25:15.586396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.586363 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xwsp\" (UniqueName: \"kubernetes.io/projected/524d8a1a-e41a-402b-a4cb-9697fc830edd-kube-api-access-5xwsp\") pod \"limitador-operator-controller-manager-859d7b78bd-q8p55\" (UID: \"524d8a1a-e41a-402b-a4cb-9697fc830edd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:15.687679 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.687642 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xwsp\" (UniqueName: \"kubernetes.io/projected/524d8a1a-e41a-402b-a4cb-9697fc830edd-kube-api-access-5xwsp\") pod \"limitador-operator-controller-manager-859d7b78bd-q8p55\" (UID: \"524d8a1a-e41a-402b-a4cb-9697fc830edd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:15.715050 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.715014 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xwsp\" (UniqueName: \"kubernetes.io/projected/524d8a1a-e41a-402b-a4cb-9697fc830edd-kube-api-access-5xwsp\") pod \"limitador-operator-controller-manager-859d7b78bd-q8p55\" (UID: \"524d8a1a-e41a-402b-a4cb-9697fc830edd\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:15.814521 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.814421 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:15.984203 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:15.984167 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55"] Jul 27 19:25:15.988963 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:15.988932 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod524d8a1a_e41a_402b_a4cb_9697fc830edd.slice/crio-9a6110392afa550a0559d920bde9c7b8f7a460120efc75cc78078392778bc533 WatchSource:0}: Error finding container 9a6110392afa550a0559d920bde9c7b8f7a460120efc75cc78078392778bc533: Status 404 returned error can't find the container with id 9a6110392afa550a0559d920bde9c7b8f7a460120efc75cc78078392778bc533 Jul 27 19:25:16.982716 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:16.982682 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" event={"ID":"524d8a1a-e41a-402b-a4cb-9697fc830edd","Type":"ContainerStarted","Data":"9a6110392afa550a0559d920bde9c7b8f7a460120efc75cc78078392778bc533"} Jul 27 19:25:17.990981 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:17.990945 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" event={"ID":"524d8a1a-e41a-402b-a4cb-9697fc830edd","Type":"ContainerStarted","Data":"d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba"} Jul 27 19:25:17.991422 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:17.991047 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:18.016105 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:18.016050 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" podStartSLOduration=1.405773986 podStartE2EDuration="3.016032219s" totalImagesPullingTime="1.610258233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:18.01329852 +0000 UTC m=+491.007795501" watchObservedRunningTime="2026-07-27 19:25:18.016032219 +0000 UTC m=+491.010529194" Jul 27 19:25:20.566806 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.566753 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9"] Jul 27 19:25:20.570525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.570504 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.573169 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.573140 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s8tnh\"" Jul 27 19:25:20.581536 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.581507 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9"] Jul 27 19:25:20.629719 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.629678 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dfa47a06-7970-4007-a3ca-94dc8736ea35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.629912 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.629779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbnsp\" (UniqueName: \"kubernetes.io/projected/dfa47a06-7970-4007-a3ca-94dc8736ea35-kube-api-access-mbnsp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.730473 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.730429 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dfa47a06-7970-4007-a3ca-94dc8736ea35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.730636 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.730542 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbnsp\" (UniqueName: \"kubernetes.io/projected/dfa47a06-7970-4007-a3ca-94dc8736ea35-kube-api-access-mbnsp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.730903 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.730880 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dfa47a06-7970-4007-a3ca-94dc8736ea35-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.743831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.743794 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbnsp\" (UniqueName: \"kubernetes.io/projected/dfa47a06-7970-4007-a3ca-94dc8736ea35-kube-api-access-mbnsp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:20.881942 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:20.881832 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:21.027576 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:21.027536 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9"] Jul 27 19:25:21.031277 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:21.031249 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa47a06_7970_4007_a3ca_94dc8736ea35.slice/crio-d11e3fae86c7a9d973af1057b89c4de7b49b6690e9a61710dc4a39592d3803c4 WatchSource:0}: Error finding container d11e3fae86c7a9d973af1057b89c4de7b49b6690e9a61710dc4a39592d3803c4: Status 404 returned error can't find the container with id d11e3fae86c7a9d973af1057b89c4de7b49b6690e9a61710dc4a39592d3803c4 Jul 27 19:25:22.014935 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:22.014889 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" event={"ID":"dfa47a06-7970-4007-a3ca-94dc8736ea35","Type":"ContainerStarted","Data":"d11e3fae86c7a9d973af1057b89c4de7b49b6690e9a61710dc4a39592d3803c4"} Jul 27 19:25:23.891209 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:23.891163 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-prpnn"] Jul 27 19:25:23.894830 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:23.894800 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:23.897250 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:23.897224 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8wfgv\"" Jul 27 19:25:23.906633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:23.906560 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-prpnn"] Jul 27 19:25:23.959860 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:23.959813 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq7vt\" (UniqueName: \"kubernetes.io/projected/ff042dda-701f-4148-96fd-ab5e1e04d6b2-kube-api-access-bq7vt\") pod \"authorino-operator-75cc8d5cb7-prpnn\" (UID: \"ff042dda-701f-4148-96fd-ab5e1e04d6b2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:24.061382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:24.061346 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq7vt\" (UniqueName: \"kubernetes.io/projected/ff042dda-701f-4148-96fd-ab5e1e04d6b2-kube-api-access-bq7vt\") pod \"authorino-operator-75cc8d5cb7-prpnn\" (UID: \"ff042dda-701f-4148-96fd-ab5e1e04d6b2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:24.070410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:24.070381 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq7vt\" (UniqueName: \"kubernetes.io/projected/ff042dda-701f-4148-96fd-ab5e1e04d6b2-kube-api-access-bq7vt\") pod \"authorino-operator-75cc8d5cb7-prpnn\" (UID: \"ff042dda-701f-4148-96fd-ab5e1e04d6b2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:24.211775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:24.211722 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:24.386557 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:24.386524 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-prpnn"] Jul 27 19:25:24.389048 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:24.389009 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff042dda_701f_4148_96fd_ab5e1e04d6b2.slice/crio-c5522d4b783366ebaa124031d265bf63ed2e989c1578358b7a91e8838cc93e9e WatchSource:0}: Error finding container c5522d4b783366ebaa124031d265bf63ed2e989c1578358b7a91e8838cc93e9e: Status 404 returned error can't find the container with id c5522d4b783366ebaa124031d265bf63ed2e989c1578358b7a91e8838cc93e9e Jul 27 19:25:25.036751 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:25.036705 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" event={"ID":"ff042dda-701f-4148-96fd-ab5e1e04d6b2","Type":"ContainerStarted","Data":"c5522d4b783366ebaa124031d265bf63ed2e989c1578358b7a91e8838cc93e9e"} Jul 27 19:25:25.980370 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:25.980093 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5wn9r" Jul 27 19:25:28.056371 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.056264 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" event={"ID":"dfa47a06-7970-4007-a3ca-94dc8736ea35","Type":"ContainerStarted","Data":"73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599"} Jul 27 19:25:28.056823 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.056446 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:28.057915 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.057889 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" event={"ID":"ff042dda-701f-4148-96fd-ab5e1e04d6b2","Type":"ContainerStarted","Data":"ca8caa2a5f4f506f334859925209c9d8f47aeafce4ef84a3fe86e3df569f29a2"} Jul 27 19:25:28.058042 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.057980 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:28.079569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.079500 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" podStartSLOduration=1.369186563 podStartE2EDuration="8.079481136s" totalImagesPullingTime="6.710294573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:28.07682938 +0000 UTC m=+501.071326356" watchObservedRunningTime="2026-07-27 19:25:28.079481136 +0000 UTC m=+501.073978111" Jul 27 19:25:28.096340 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.096274 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" podStartSLOduration=1.706398028 podStartE2EDuration="5.096254116s" totalImagesPullingTime="3.389856088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:28.094149722 +0000 UTC m=+501.088646697" watchObservedRunningTime="2026-07-27 19:25:28.096254116 +0000 UTC m=+501.090751170" Jul 27 19:25:28.998493 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:28.998462 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:30.682474 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.682435 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh"] Jul 27 19:25:30.686095 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.686072 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.688500 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.688476 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-6w68x\"" Jul 27 19:25:30.688500 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.688474 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 27 19:25:30.688696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.688476 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 27 19:25:30.696399 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.696370 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh"] Jul 27 19:25:30.829397 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.829350 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ee52f17c-b2db-4b48-9058-28f205a9de33-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.829397 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.829390 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsm7\" (UniqueName: \"kubernetes.io/projected/ee52f17c-b2db-4b48-9058-28f205a9de33-kube-api-access-6bsm7\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.829611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.829471 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee52f17c-b2db-4b48-9058-28f205a9de33-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.930261 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.930214 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ee52f17c-b2db-4b48-9058-28f205a9de33-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.930261 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.930266 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsm7\" (UniqueName: \"kubernetes.io/projected/ee52f17c-b2db-4b48-9058-28f205a9de33-kube-api-access-6bsm7\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.930568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.930310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee52f17c-b2db-4b48-9058-28f205a9de33-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.930568 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:30.930487 2566 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 27 19:25:30.930660 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:30.930573 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee52f17c-b2db-4b48-9058-28f205a9de33-plugin-serving-cert podName:ee52f17c-b2db-4b48-9058-28f205a9de33 nodeName:}" failed. No retries permitted until 2026-07-27 19:25:31.430543947 +0000 UTC m=+504.425040905 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/ee52f17c-b2db-4b48-9058-28f205a9de33-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-k9wlh" (UID: "ee52f17c-b2db-4b48-9058-28f205a9de33") : secret "plugin-serving-cert" not found Jul 27 19:25:30.931012 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.930989 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ee52f17c-b2db-4b48-9058-28f205a9de33-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:30.952433 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:30.952342 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsm7\" (UniqueName: \"kubernetes.io/projected/ee52f17c-b2db-4b48-9058-28f205a9de33-kube-api-access-6bsm7\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:31.435043 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:31.435010 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee52f17c-b2db-4b48-9058-28f205a9de33-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:31.437522 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:31.437500 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/ee52f17c-b2db-4b48-9058-28f205a9de33-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-k9wlh\" (UID: \"ee52f17c-b2db-4b48-9058-28f205a9de33\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:31.596960 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:31.596906 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" Jul 27 19:25:31.744201 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:31.744172 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh"] Jul 27 19:25:31.746989 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:31.746950 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee52f17c_b2db_4b48_9058_28f205a9de33.slice/crio-38154f993462f119bbf9d59a9e9e149d57dd6092a4516a1e3761b6645f56a27b WatchSource:0}: Error finding container 38154f993462f119bbf9d59a9e9e149d57dd6092a4516a1e3761b6645f56a27b: Status 404 returned error can't find the container with id 38154f993462f119bbf9d59a9e9e149d57dd6092a4516a1e3761b6645f56a27b Jul 27 19:25:32.080048 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:32.079966 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" event={"ID":"ee52f17c-b2db-4b48-9058-28f205a9de33","Type":"ContainerStarted","Data":"38154f993462f119bbf9d59a9e9e149d57dd6092a4516a1e3761b6645f56a27b"} Jul 27 19:25:39.065631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:39.065601 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:39.066193 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:39.065673 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-prpnn" Jul 27 19:25:40.391943 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.391906 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g"] Jul 27 19:25:40.410183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.410149 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.415250 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.415175 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g"] Jul 27 19:25:40.432094 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.432060 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/530659e3-757e-48fe-9e45-124c780a2089-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.432276 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.432126 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4qs\" (UniqueName: \"kubernetes.io/projected/530659e3-757e-48fe-9e45-124c780a2089-kube-api-access-lm4qs\") pod \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.532847 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.532803 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/530659e3-757e-48fe-9e45-124c780a2089-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.533047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.532896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4qs\" (UniqueName: \"kubernetes.io/projected/530659e3-757e-48fe-9e45-124c780a2089-kube-api-access-lm4qs\") pod \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.533303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.533277 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/530659e3-757e-48fe-9e45-124c780a2089-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.541982 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.541953 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4qs\" (UniqueName: \"kubernetes.io/projected/530659e3-757e-48fe-9e45-124c780a2089-kube-api-access-lm4qs\") pod \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.688888 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.688849 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g"] Jul 27 19:25:40.689246 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.689224 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:40.693252 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.693220 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g"] Jul 27 19:25:40.706781 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.706732 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9"] Jul 27 19:25:40.707051 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.706980 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" containerName="manager" containerID="cri-o://73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599" gracePeriod=2 Jul 27 19:25:40.715532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.715493 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp"] Jul 27 19:25:40.730688 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:40.730632 2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"dfa47a06-7970-4007-a3ca-94dc8736ea35\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\": pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:40.735015 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.734977 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9"] Jul 27 19:25:40.735184 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.735123 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:40.738083 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.738052 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp"] Jul 27 19:25:40.747272 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.747203 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw"] Jul 27 19:25:40.748512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.748484 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" containerName="manager" Jul 27 19:25:40.748693 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.748682 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" containerName="manager" Jul 27 19:25:40.749054 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.749038 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" containerName="manager" Jul 27 19:25:40.754537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.754509 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55"] Jul 27 19:25:40.754849 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.754830 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55"] Jul 27 19:25:40.756162 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.756139 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.756573 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.756065 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" containerName="manager" containerID="cri-o://d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba" gracePeriod=2 Jul 27 19:25:40.758331 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.758140 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw"] Jul 27 19:25:40.764410 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:40.764362 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:40.795597 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:40.795543 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8p55\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:40.797277 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:40.797234 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:40.837486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.837448 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntkp7\" (UniqueName: \"kubernetes.io/projected/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-kube-api-access-ntkp7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mf7jw\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.837486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.837502 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzq6z\" (UniqueName: \"kubernetes.io/projected/61a6155f-9192-46f9-9ab0-53c2dd7ed390-kube-api-access-wzq6z\") pod \"kuadrant-operator-controller-manager-55467d9f7-ctfbp\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:40.837784 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.837566 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mf7jw\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.837784 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.837675 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61a6155f-9192-46f9-9ab0-53c2dd7ed390-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-ctfbp\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:40.938880 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.938837 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntkp7\" (UniqueName: \"kubernetes.io/projected/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-kube-api-access-ntkp7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mf7jw\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.939116 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.938896 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzq6z\" (UniqueName: \"kubernetes.io/projected/61a6155f-9192-46f9-9ab0-53c2dd7ed390-kube-api-access-wzq6z\") pod \"kuadrant-operator-controller-manager-55467d9f7-ctfbp\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:40.939116 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.938954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mf7jw\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.939116 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.938997 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61a6155f-9192-46f9-9ab0-53c2dd7ed390-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-ctfbp\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:40.939525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.939495 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mf7jw\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.939660 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.939497 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61a6155f-9192-46f9-9ab0-53c2dd7ed390-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-55467d9f7-ctfbp\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:40.948337 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.948300 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntkp7\" (UniqueName: \"kubernetes.io/projected/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-kube-api-access-ntkp7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-mf7jw\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:40.948512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:40.948301 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzq6z\" (UniqueName: \"kubernetes.io/projected/61a6155f-9192-46f9-9ab0-53c2dd7ed390-kube-api-access-wzq6z\") pod \"kuadrant-operator-controller-manager-55467d9f7-ctfbp\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:41.135833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:41.135778 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:41.143609 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:41.143566 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:53.583507 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.583473 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:53.587909 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:53.587870 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:53.602255 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.602154 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:53.604479 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:53.604443 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:53.607859 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:53.607820 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8p55\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:53.610143 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.610116 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw"] Jul 27 19:25:53.613310 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:53.613276 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc532c631_8dc0_4696_9d41_e69b4e7c3c0b.slice/crio-1c0fb27d79d3ab6adfb1201c90ff02c4241115435e43077f21f927ac7fef02f2 WatchSource:0}: Error finding container 1c0fb27d79d3ab6adfb1201c90ff02c4241115435e43077f21f927ac7fef02f2: Status 404 returned error can't find the container with id 1c0fb27d79d3ab6adfb1201c90ff02c4241115435e43077f21f927ac7fef02f2 Jul 27 19:25:53.642275 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.642208 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp"] Jul 27 19:25:53.648063 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:25:53.648031 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a6155f_9192_46f9_9ab0_53c2dd7ed390.slice/crio-8abc201a120659231c839756b3d92d7ddd7c0dd2ba213df31878ac745eadbe68 WatchSource:0}: Error finding container 8abc201a120659231c839756b3d92d7ddd7c0dd2ba213df31878ac745eadbe68: Status 404 returned error can't find the container with id 8abc201a120659231c839756b3d92d7ddd7c0dd2ba213df31878ac745eadbe68 Jul 27 19:25:53.673065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.673034 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mbnsp\" (UniqueName: \"kubernetes.io/projected/dfa47a06-7970-4007-a3ca-94dc8736ea35-kube-api-access-mbnsp\") pod \"dfa47a06-7970-4007-a3ca-94dc8736ea35\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " Jul 27 19:25:53.673303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.673279 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dfa47a06-7970-4007-a3ca-94dc8736ea35-extensions-socket-volume\") pod \"dfa47a06-7970-4007-a3ca-94dc8736ea35\" (UID: \"dfa47a06-7970-4007-a3ca-94dc8736ea35\") " Jul 27 19:25:53.673388 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.673340 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xwsp\" (UniqueName: \"kubernetes.io/projected/524d8a1a-e41a-402b-a4cb-9697fc830edd-kube-api-access-5xwsp\") pod \"524d8a1a-e41a-402b-a4cb-9697fc830edd\" (UID: \"524d8a1a-e41a-402b-a4cb-9697fc830edd\") " Jul 27 19:25:53.674452 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.674402 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa47a06-7970-4007-a3ca-94dc8736ea35-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dfa47a06-7970-4007-a3ca-94dc8736ea35" (UID: "dfa47a06-7970-4007-a3ca-94dc8736ea35"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:25:53.675643 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.675616 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa47a06-7970-4007-a3ca-94dc8736ea35-kube-api-access-mbnsp" (OuterVolumeSpecName: "kube-api-access-mbnsp") pod "dfa47a06-7970-4007-a3ca-94dc8736ea35" (UID: "dfa47a06-7970-4007-a3ca-94dc8736ea35"). InnerVolumeSpecName "kube-api-access-mbnsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:25:53.675824 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.675803 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/524d8a1a-e41a-402b-a4cb-9697fc830edd-kube-api-access-5xwsp" (OuterVolumeSpecName: "kube-api-access-5xwsp") pod "524d8a1a-e41a-402b-a4cb-9697fc830edd" (UID: "524d8a1a-e41a-402b-a4cb-9697fc830edd"). InnerVolumeSpecName "kube-api-access-5xwsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:25:53.774905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.774861 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dfa47a06-7970-4007-a3ca-94dc8736ea35-extensions-socket-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:25:53.774905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.774895 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xwsp\" (UniqueName: \"kubernetes.io/projected/524d8a1a-e41a-402b-a4cb-9697fc830edd-kube-api-access-5xwsp\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:25:53.774905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:53.774905 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbnsp\" (UniqueName: \"kubernetes.io/projected/dfa47a06-7970-4007-a3ca-94dc8736ea35-kube-api-access-mbnsp\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:25:54.227657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.227617 2566 generic.go:356] "Generic (PLEG): container finished" podID="524d8a1a-e41a-402b-a4cb-9697fc830edd" containerID="d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba" exitCode=0 Jul 27 19:25:54.227657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.227662 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:54.227901 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.227725 2566 scope.go:117] "RemoveContainer" containerID="d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba" Jul 27 19:25:54.229489 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.229456 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" event={"ID":"ee52f17c-b2db-4b48-9058-28f205a9de33","Type":"ContainerStarted","Data":"55be99c34226b6a7af74bca898344d4422bd0bf4306d92a4516aefc17c418212"} Jul 27 19:25:54.231407 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.231371 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" event={"ID":"c532c631-8dc0-4696-9d41-e69b4e7c3c0b","Type":"ContainerStarted","Data":"5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123"} Jul 27 19:25:54.231560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.231428 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" event={"ID":"c532c631-8dc0-4696-9d41-e69b4e7c3c0b","Type":"ContainerStarted","Data":"1c0fb27d79d3ab6adfb1201c90ff02c4241115435e43077f21f927ac7fef02f2"} Jul 27 19:25:54.231560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.231467 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:25:54.232672 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.232640 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:54.232783 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.232735 2566 generic.go:356] "Generic (PLEG): container finished" podID="dfa47a06-7970-4007-a3ca-94dc8736ea35" containerID="73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599" exitCode=0 Jul 27 19:25:54.232858 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.232801 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:54.234631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.234598 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" event={"ID":"61a6155f-9192-46f9-9ab0-53c2dd7ed390","Type":"ContainerStarted","Data":"49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d"} Jul 27 19:25:54.234734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.234646 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" event={"ID":"61a6155f-9192-46f9-9ab0-53c2dd7ed390","Type":"ContainerStarted","Data":"8abc201a120659231c839756b3d92d7ddd7c0dd2ba213df31878ac745eadbe68"} Jul 27 19:25:54.234734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.234677 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:25:54.235451 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.235414 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8p55\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:54.238467 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.238429 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:54.238941 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.238917 2566 scope.go:117] "RemoveContainer" containerID="d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba" Jul 27 19:25:54.239242 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.239223 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba\": container with ID starting with d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba not found: ID does not exist" containerID="d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba" Jul 27 19:25:54.239327 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.239256 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba"} err="failed to get container status \"d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba\": rpc error: code = NotFound desc = could not find container \"d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba\": container with ID starting with d362cc67e095de456ea7d8b2b12f9cff94696399ed04b80362f53237b30610ba not found: ID does not exist" Jul 27 19:25:54.239327 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.239283 2566 scope.go:117] "RemoveContainer" containerID="73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599" Jul 27 19:25:54.240704 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.240676 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8p55\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:54.248242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.248216 2566 scope.go:117] "RemoveContainer" containerID="73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599" Jul 27 19:25:54.248547 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.248528 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599\": container with ID starting with 73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599 not found: ID does not exist" containerID="73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599" Jul 27 19:25:54.248591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.248557 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599"} err="failed to get container status \"73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599\": rpc error: code = NotFound desc = could not find container \"73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599\": container with ID starting with 73d75c8e923d9fe4968891672c6eabe6bd1819bc221bad5a57680c35b0ef4599 not found: ID does not exist" Jul 27 19:25:54.272986 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.272925 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" podStartSLOduration=14.272908555 podStartE2EDuration="14.272908555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:54.27018747 +0000 UTC m=+527.264684447" watchObservedRunningTime="2026-07-27 19:25:54.272908555 +0000 UTC m=+527.267405523" Jul 27 19:25:54.315709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.315616 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-k9wlh" podStartSLOduration=2.558789642 podStartE2EDuration="24.315596144s" totalImagesPullingTime="21.756806502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:54.315027098 +0000 UTC m=+527.309524075" watchObservedRunningTime="2026-07-27 19:25:54.315596144 +0000 UTC m=+527.310093119" Jul 27 19:25:54.348956 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.348910 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-7rjc9\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-7rjc9" Jul 27 19:25:54.348956 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:54.348934 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" podStartSLOduration=14.348918592 podStartE2EDuration="14.348918592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:25:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:25:54.346678026 +0000 UTC m=+527.341175001" watchObservedRunningTime="2026-07-27 19:25:54.348918592 +0000 UTC m=+527.343415567" Jul 27 19:25:54.351301 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:54.351275 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-q8p55\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-q8p55" Jul 27 19:25:55.546444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:55.546408 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" path="/var/lib/kubelet/pods/524d8a1a-e41a-402b-a4cb-9697fc830edd/volumes" Jul 27 19:25:55.546879 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:55.546861 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfa47a06-7970-4007-a3ca-94dc8736ea35" path="/var/lib/kubelet/pods/dfa47a06-7970-4007-a3ca-94dc8736ea35/volumes" Jul 27 19:25:55.946226 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:55.946185 2566 log.go:32] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-x5z7g_kuadrant-system_530659e3-757e-48fe-9e45-124c780a2089_0(741cf06819aae3c5db28381df26db5ab4f78c14188d2333d686695d3fec07378): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-x5z7g to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" Jul 27 19:25:55.946299 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:55.946254 2566 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-55467d9f7-x5z7g_kuadrant-system_530659e3-757e-48fe-9e45-124c780a2089_0(741cf06819aae3c5db28381df26db5ab4f78c14188d2333d686695d3fec07378): error adding pod kuadrant-system_kuadrant-operator-controller-manager-55467d9f7-x5z7g to CNI network \"multus-cni-network\": plugin type=\"multus-shim\" name=\"multus-cni-network\" failed (add): CmdAdd (shim): failed to send CNI request: Post \"http://dummy/cni\": EOF: StdinData: {\"auxiliaryCNIChainName\":\"vendor-cni-chain\",\"binDir\":\"/var/lib/cni/bin\",\"clusterNetwork\":\"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf\",\"cniVersion\":\"0.3.1\",\"daemonSocketDir\":\"/run/multus/socket\",\"globalNamespaces\":\"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv\",\"logLevel\":\"verbose\",\"logToStderr\":true,\"name\":\"multus-cni-network\",\"namespaceIsolation\":true,\"type\":\"multus-shim\"}" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:56.247532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.247430 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:56.250145 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:56.250115 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="530659e3-757e-48fe-9e45-124c780a2089" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:56.256336 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.256308 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:56.258519 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:56.258488 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="530659e3-757e-48fe-9e45-124c780a2089" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:56.399610 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.399575 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/530659e3-757e-48fe-9e45-124c780a2089-extensions-socket-volume\") pod \"530659e3-757e-48fe-9e45-124c780a2089\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " Jul 27 19:25:56.399856 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.399659 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm4qs\" (UniqueName: \"kubernetes.io/projected/530659e3-757e-48fe-9e45-124c780a2089-kube-api-access-lm4qs\") pod \"530659e3-757e-48fe-9e45-124c780a2089\" (UID: \"530659e3-757e-48fe-9e45-124c780a2089\") " Jul 27 19:25:56.399962 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.399937 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/530659e3-757e-48fe-9e45-124c780a2089-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "530659e3-757e-48fe-9e45-124c780a2089" (UID: "530659e3-757e-48fe-9e45-124c780a2089"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:25:56.401863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.401832 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/530659e3-757e-48fe-9e45-124c780a2089-kube-api-access-lm4qs" (OuterVolumeSpecName: "kube-api-access-lm4qs") pod "530659e3-757e-48fe-9e45-124c780a2089" (UID: "530659e3-757e-48fe-9e45-124c780a2089"). InnerVolumeSpecName "kube-api-access-lm4qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:25:56.500974 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.500874 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/530659e3-757e-48fe-9e45-124c780a2089-extensions-socket-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:25:56.500974 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:56.500910 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4qs\" (UniqueName: \"kubernetes.io/projected/530659e3-757e-48fe-9e45-124c780a2089-kube-api-access-lm4qs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:25:57.253698 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:57.253662 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:57.255998 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:57.255964 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="530659e3-757e-48fe-9e45-124c780a2089" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:57.262185 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:57.262149 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="530659e3-757e-48fe-9e45-124c780a2089" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:57.544218 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:25:57.544123 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-55467d9f7-x5z7g\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="530659e3-757e-48fe-9e45-124c780a2089" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-x5z7g" Jul 27 19:25:57.546404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:25:57.546378 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="530659e3-757e-48fe-9e45-124c780a2089" path="/var/lib/kubelet/pods/530659e3-757e-48fe-9e45-124c780a2089/volumes" Jul 27 19:26:05.244300 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:05.244212 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:26:05.244807 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:05.244315 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:26:33.361494 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.361458 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw"] Jul 27 19:26:33.362053 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.361708 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" podUID="c532c631-8dc0-4696-9d41-e69b4e7c3c0b" containerName="manager" containerID="cri-o://5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123" gracePeriod=10 Jul 27 19:26:33.623900 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.623821 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:26:33.695192 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695157 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk"] Jul 27 19:26:33.695585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695572 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" containerName="manager" Jul 27 19:26:33.695633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695587 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" containerName="manager" Jul 27 19:26:33.695633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695598 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c532c631-8dc0-4696-9d41-e69b4e7c3c0b" containerName="manager" Jul 27 19:26:33.695633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695604 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c532c631-8dc0-4696-9d41-e69b4e7c3c0b" containerName="manager" Jul 27 19:26:33.695726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695669 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c532c631-8dc0-4696-9d41-e69b4e7c3c0b" containerName="manager" Jul 27 19:26:33.695726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.695678 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="524d8a1a-e41a-402b-a4cb-9697fc830edd" containerName="manager" Jul 27 19:26:33.698749 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.698733 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:33.729800 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.729755 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk"] Jul 27 19:26:33.740003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.739970 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntkp7\" (UniqueName: \"kubernetes.io/projected/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-kube-api-access-ntkp7\") pod \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " Jul 27 19:26:33.740197 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.740051 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-extensions-socket-volume\") pod \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\" (UID: \"c532c631-8dc0-4696-9d41-e69b4e7c3c0b\") " Jul 27 19:26:33.740266 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.740193 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhglf\" (UniqueName: \"kubernetes.io/projected/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-kube-api-access-nhglf\") pod \"kuadrant-operator-controller-manager-86878c769f-47vpk\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:33.740321 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.740294 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47vpk\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:33.740495 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.740463 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c532c631-8dc0-4696-9d41-e69b4e7c3c0b" (UID: "c532c631-8dc0-4696-9d41-e69b4e7c3c0b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:33.742364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.742334 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-kube-api-access-ntkp7" (OuterVolumeSpecName: "kube-api-access-ntkp7") pod "c532c631-8dc0-4696-9d41-e69b4e7c3c0b" (UID: "c532c631-8dc0-4696-9d41-e69b4e7c3c0b"). InnerVolumeSpecName "kube-api-access-ntkp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:26:33.841122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.841077 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhglf\" (UniqueName: \"kubernetes.io/projected/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-kube-api-access-nhglf\") pod \"kuadrant-operator-controller-manager-86878c769f-47vpk\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:33.841313 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.841165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47vpk\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:33.841313 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.841210 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntkp7\" (UniqueName: \"kubernetes.io/projected/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-kube-api-access-ntkp7\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:26:33.841313 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.841223 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c532c631-8dc0-4696-9d41-e69b4e7c3c0b-extensions-socket-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:26:33.841557 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.841539 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-47vpk\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:33.851566 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:33.851533 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhglf\" (UniqueName: \"kubernetes.io/projected/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-kube-api-access-nhglf\") pod \"kuadrant-operator-controller-manager-86878c769f-47vpk\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:34.009720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.009682 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:34.164703 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.164662 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk"] Jul 27 19:26:34.168021 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:26:34.167994 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b0e406_5f19_4aea_a13c_c6a5bcb0b83e.slice/crio-22867caf18e2330a5c3df95252efcd37f3d36193cd3e0ba32ac4980ea11dfa30 WatchSource:0}: Error finding container 22867caf18e2330a5c3df95252efcd37f3d36193cd3e0ba32ac4980ea11dfa30: Status 404 returned error can't find the container with id 22867caf18e2330a5c3df95252efcd37f3d36193cd3e0ba32ac4980ea11dfa30 Jul 27 19:26:34.450223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.450186 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" event={"ID":"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e","Type":"ContainerStarted","Data":"548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f"} Jul 27 19:26:34.450672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.450249 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:34.450672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.450271 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" event={"ID":"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e","Type":"ContainerStarted","Data":"22867caf18e2330a5c3df95252efcd37f3d36193cd3e0ba32ac4980ea11dfa30"} Jul 27 19:26:34.451598 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.451573 2566 generic.go:356] "Generic (PLEG): container finished" podID="c532c631-8dc0-4696-9d41-e69b4e7c3c0b" containerID="5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123" exitCode=0 Jul 27 19:26:34.451664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.451615 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" event={"ID":"c532c631-8dc0-4696-9d41-e69b4e7c3c0b","Type":"ContainerDied","Data":"5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123"} Jul 27 19:26:34.451664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.451651 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" event={"ID":"c532c631-8dc0-4696-9d41-e69b4e7c3c0b","Type":"ContainerDied","Data":"1c0fb27d79d3ab6adfb1201c90ff02c4241115435e43077f21f927ac7fef02f2"} Jul 27 19:26:34.451743 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.451674 2566 scope.go:117] "RemoveContainer" containerID="5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123" Jul 27 19:26:34.451743 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.451618 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw" Jul 27 19:26:34.461309 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.461288 2566 scope.go:117] "RemoveContainer" containerID="5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123" Jul 27 19:26:34.461621 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:26:34.461602 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123\": container with ID starting with 5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123 not found: ID does not exist" containerID="5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123" Jul 27 19:26:34.461692 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.461637 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123"} err="failed to get container status \"5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123\": rpc error: code = NotFound desc = could not find container \"5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123\": container with ID starting with 5115703c2090c6c95a425e45a81d8a0a5689dd6a237d6b7c7da12d161d1ef123 not found: ID does not exist" Jul 27 19:26:34.474199 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.474147 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" podStartSLOduration=1.474131646 podStartE2EDuration="1.474131646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:34.471738802 +0000 UTC m=+567.466235776" watchObservedRunningTime="2026-07-27 19:26:34.474131646 +0000 UTC m=+567.468628620" Jul 27 19:26:34.491373 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.491338 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw"] Jul 27 19:26:34.505723 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:34.505686 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-mf7jw"] Jul 27 19:26:35.545886 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:35.545853 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c532c631-8dc0-4696-9d41-e69b4e7c3c0b" path="/var/lib/kubelet/pods/c532c631-8dc0-4696-9d41-e69b4e7c3c0b/volumes" Jul 27 19:26:45.462005 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.461970 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:26:45.555153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.555119 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp"] Jul 27 19:26:45.555337 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.555316 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" podUID="61a6155f-9192-46f9-9ab0-53c2dd7ed390" containerName="manager" containerID="cri-o://49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d" gracePeriod=10 Jul 27 19:26:45.823778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.823743 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:26:45.956160 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.956119 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzq6z\" (UniqueName: \"kubernetes.io/projected/61a6155f-9192-46f9-9ab0-53c2dd7ed390-kube-api-access-wzq6z\") pod \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " Jul 27 19:26:45.956358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.956230 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61a6155f-9192-46f9-9ab0-53c2dd7ed390-extensions-socket-volume\") pod \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\" (UID: \"61a6155f-9192-46f9-9ab0-53c2dd7ed390\") " Jul 27 19:26:45.956646 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.956621 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61a6155f-9192-46f9-9ab0-53c2dd7ed390-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "61a6155f-9192-46f9-9ab0-53c2dd7ed390" (UID: "61a6155f-9192-46f9-9ab0-53c2dd7ed390"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:26:45.958347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:45.958309 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61a6155f-9192-46f9-9ab0-53c2dd7ed390-kube-api-access-wzq6z" (OuterVolumeSpecName: "kube-api-access-wzq6z") pod "61a6155f-9192-46f9-9ab0-53c2dd7ed390" (UID: "61a6155f-9192-46f9-9ab0-53c2dd7ed390"). InnerVolumeSpecName "kube-api-access-wzq6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:26:46.057413 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.057320 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/61a6155f-9192-46f9-9ab0-53c2dd7ed390-extensions-socket-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:26:46.057413 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.057351 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzq6z\" (UniqueName: \"kubernetes.io/projected/61a6155f-9192-46f9-9ab0-53c2dd7ed390-kube-api-access-wzq6z\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:26:46.523120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.523086 2566 generic.go:356] "Generic (PLEG): container finished" podID="61a6155f-9192-46f9-9ab0-53c2dd7ed390" containerID="49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d" exitCode=0 Jul 27 19:26:46.523571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.523128 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" Jul 27 19:26:46.523571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.523171 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" event={"ID":"61a6155f-9192-46f9-9ab0-53c2dd7ed390","Type":"ContainerDied","Data":"49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d"} Jul 27 19:26:46.523571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.523220 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp" event={"ID":"61a6155f-9192-46f9-9ab0-53c2dd7ed390","Type":"ContainerDied","Data":"8abc201a120659231c839756b3d92d7ddd7c0dd2ba213df31878ac745eadbe68"} Jul 27 19:26:46.523571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.523241 2566 scope.go:117] "RemoveContainer" containerID="49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d" Jul 27 19:26:46.532687 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.532666 2566 scope.go:117] "RemoveContainer" containerID="49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d" Jul 27 19:26:46.533011 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:26:46.532986 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d\": container with ID starting with 49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d not found: ID does not exist" containerID="49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d" Jul 27 19:26:46.533087 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.533024 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d"} err="failed to get container status \"49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d\": rpc error: code = NotFound desc = could not find container \"49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d\": container with ID starting with 49761321cf3130b2562a07181ec0ebfeb1370061036cb7582c834df7fef5cf4d not found: ID does not exist" Jul 27 19:26:46.552270 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.552229 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp"] Jul 27 19:26:46.559891 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:46.559862 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-55467d9f7-ctfbp"] Jul 27 19:26:47.546497 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:47.546407 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61a6155f-9192-46f9-9ab0-53c2dd7ed390" path="/var/lib/kubelet/pods/61a6155f-9192-46f9-9ab0-53c2dd7ed390/volumes" Jul 27 19:26:49.864284 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.864248 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs"] Jul 27 19:26:49.864691 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.864677 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61a6155f-9192-46f9-9ab0-53c2dd7ed390" containerName="manager" Jul 27 19:26:49.864734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.864693 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="61a6155f-9192-46f9-9ab0-53c2dd7ed390" containerName="manager" Jul 27 19:26:49.864789 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.864775 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="61a6155f-9192-46f9-9ab0-53c2dd7ed390" containerName="manager" Jul 27 19:26:49.869166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.869142 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:49.871685 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.871659 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-7dtb8\"" Jul 27 19:26:49.884894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:49.884859 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs"] Jul 27 19:26:50.001230 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001190 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001230 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001230 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001292 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b88wh\" (UniqueName: \"kubernetes.io/projected/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-kube-api-access-b88wh\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001391 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001415 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001612 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001467 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001612 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.001612 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.001518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.102294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.102294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102294 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.102532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102317 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.102532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102340 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b88wh\" (UniqueName: \"kubernetes.io/projected/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-kube-api-access-b88wh\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.102532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102368 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.102532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102393 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102570 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.102774 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-data\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103269 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.103162 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103517 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.103299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-credential-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103517 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.103407 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-workload-socket\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.103517 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.103505 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-workload-certs\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.105095 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.105075 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.105223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.105207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.124736 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.124657 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-istio-token\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.127615 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.127586 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b88wh\" (UniqueName: \"kubernetes.io/projected/61b8a2c6-049e-4451-9c7d-7a0670d98ba1-kube-api-access-b88wh\") pod \"maas-default-gateway-openshift-default-5ccb968c66-tqprs\" (UID: \"61b8a2c6-049e-4451-9c7d-7a0670d98ba1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.179479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.179439 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:50.326004 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.325977 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs"] Jul 27 19:26:50.328533 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:26:50.328510 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61b8a2c6_049e_4451_9c7d_7a0670d98ba1.slice/crio-83bfba4f5223f60b20b8a5e38d70ea2e8aa8b05d25a7a534b7e9a982ab982bba WatchSource:0}: Error finding container 83bfba4f5223f60b20b8a5e38d70ea2e8aa8b05d25a7a534b7e9a982ab982bba: Status 404 returned error can't find the container with id 83bfba4f5223f60b20b8a5e38d70ea2e8aa8b05d25a7a534b7e9a982ab982bba Jul 27 19:26:50.330550 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.330521 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:26:50.330619 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.330588 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:26:50.330657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.330618 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:26:50.559279 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.559239 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" event={"ID":"61b8a2c6-049e-4451-9c7d-7a0670d98ba1","Type":"ContainerStarted","Data":"8b91a177e6c751ab83179ce409b85d4340fe56376faf0d7604d81a19409fa014"} Jul 27 19:26:50.559471 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.559332 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" event={"ID":"61b8a2c6-049e-4451-9c7d-7a0670d98ba1","Type":"ContainerStarted","Data":"83bfba4f5223f60b20b8a5e38d70ea2e8aa8b05d25a7a534b7e9a982ab982bba"} Jul 27 19:26:50.623874 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:50.623820 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" podStartSLOduration=1.623801645 podStartE2EDuration="1.623801645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:50.620729833 +0000 UTC m=+583.615226810" watchObservedRunningTime="2026-07-27 19:26:50.623801645 +0000 UTC m=+583.618298620" Jul 27 19:26:51.180372 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:51.180339 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:51.185652 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:51.185623 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:51.564568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:51.564477 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:51.565735 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:51.565712 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:26:52.396951 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.396908 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:26:52.401028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.401007 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.403521 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.403502 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 27 19:26:52.411263 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.411236 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:26:52.491299 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.491267 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:26:52.528162 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.528123 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d55f8dc-da6e-4770-9412-1e79ec7badd6-config-file\") pod \"limitador-limitador-64c94fd64f-jfvdl\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.528348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.528195 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svb6c\" (UniqueName: \"kubernetes.io/projected/7d55f8dc-da6e-4770-9412-1e79ec7badd6-kube-api-access-svb6c\") pod \"limitador-limitador-64c94fd64f-jfvdl\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.629539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.629507 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d55f8dc-da6e-4770-9412-1e79ec7badd6-config-file\") pod \"limitador-limitador-64c94fd64f-jfvdl\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.629715 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.629563 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svb6c\" (UniqueName: \"kubernetes.io/projected/7d55f8dc-da6e-4770-9412-1e79ec7badd6-kube-api-access-svb6c\") pod \"limitador-limitador-64c94fd64f-jfvdl\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.630212 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.630192 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d55f8dc-da6e-4770-9412-1e79ec7badd6-config-file\") pod \"limitador-limitador-64c94fd64f-jfvdl\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.639019 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.638992 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-svb6c\" (UniqueName: \"kubernetes.io/projected/7d55f8dc-da6e-4770-9412-1e79ec7badd6-kube-api-access-svb6c\") pod \"limitador-limitador-64c94fd64f-jfvdl\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.712725 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.712683 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:52.876167 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:52.876132 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:26:52.877871 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:26:52.877840 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d55f8dc_da6e_4770_9412_1e79ec7badd6.slice/crio-620cacdcc66adbbc7b819151a8ef4419bd7e7cc8b95f7105969759dd300666d7 WatchSource:0}: Error finding container 620cacdcc66adbbc7b819151a8ef4419bd7e7cc8b95f7105969759dd300666d7: Status 404 returned error can't find the container with id 620cacdcc66adbbc7b819151a8ef4419bd7e7cc8b95f7105969759dd300666d7 Jul 27 19:26:53.576429 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:53.576388 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" event={"ID":"7d55f8dc-da6e-4770-9412-1e79ec7badd6","Type":"ContainerStarted","Data":"620cacdcc66adbbc7b819151a8ef4419bd7e7cc8b95f7105969759dd300666d7"} Jul 27 19:26:56.595120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:56.595079 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" event={"ID":"7d55f8dc-da6e-4770-9412-1e79ec7badd6","Type":"ContainerStarted","Data":"bc414c1931f8c13edb654aa7d8111413d6e2ca1adad9e23467271ce54b344218"} Jul 27 19:26:56.595563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:56.595222 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:26:56.617199 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:26:56.617140 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" podStartSLOduration=1.83962565 podStartE2EDuration="4.617124836s" totalImagesPullingTime="2.777499186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:26:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:26:56.616332309 +0000 UTC m=+589.610829285" watchObservedRunningTime="2026-07-27 19:26:56.617124836 +0000 UTC m=+589.611621811" Jul 27 19:27:07.048099 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.048055 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:27:07.048721 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.048288 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" podUID="7d55f8dc-da6e-4770-9412-1e79ec7badd6" containerName="limitador" containerID="cri-o://bc414c1931f8c13edb654aa7d8111413d6e2ca1adad9e23467271ce54b344218" gracePeriod=30 Jul 27 19:27:07.049065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.048926 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:27:07.485305 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.485277 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:27:07.490780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.490738 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:27:07.652737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.652648 2566 generic.go:356] "Generic (PLEG): container finished" podID="7d55f8dc-da6e-4770-9412-1e79ec7badd6" containerID="bc414c1931f8c13edb654aa7d8111413d6e2ca1adad9e23467271ce54b344218" exitCode=0 Jul 27 19:27:07.652737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:07.652693 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" event={"ID":"7d55f8dc-da6e-4770-9412-1e79ec7badd6","Type":"ContainerDied","Data":"bc414c1931f8c13edb654aa7d8111413d6e2ca1adad9e23467271ce54b344218"} Jul 27 19:27:08.010885 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.010855 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:27:08.066537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.066498 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d55f8dc-da6e-4770-9412-1e79ec7badd6-config-file\") pod \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " Jul 27 19:27:08.066965 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.066571 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-svb6c\" (UniqueName: \"kubernetes.io/projected/7d55f8dc-da6e-4770-9412-1e79ec7badd6-kube-api-access-svb6c\") pod \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\" (UID: \"7d55f8dc-da6e-4770-9412-1e79ec7badd6\") " Jul 27 19:27:08.066965 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.066918 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d55f8dc-da6e-4770-9412-1e79ec7badd6-config-file" (OuterVolumeSpecName: "config-file") pod "7d55f8dc-da6e-4770-9412-1e79ec7badd6" (UID: "7d55f8dc-da6e-4770-9412-1e79ec7badd6"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:27:08.068706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.068675 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d55f8dc-da6e-4770-9412-1e79ec7badd6-kube-api-access-svb6c" (OuterVolumeSpecName: "kube-api-access-svb6c") pod "7d55f8dc-da6e-4770-9412-1e79ec7badd6" (UID: "7d55f8dc-da6e-4770-9412-1e79ec7badd6"). InnerVolumeSpecName "kube-api-access-svb6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:27:08.168310 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.168201 2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7d55f8dc-da6e-4770-9412-1e79ec7badd6-config-file\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:27:08.168310 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.168246 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svb6c\" (UniqueName: \"kubernetes.io/projected/7d55f8dc-da6e-4770-9412-1e79ec7badd6-kube-api-access-svb6c\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:27:08.659364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.659304 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" event={"ID":"7d55f8dc-da6e-4770-9412-1e79ec7badd6","Type":"ContainerDied","Data":"620cacdcc66adbbc7b819151a8ef4419bd7e7cc8b95f7105969759dd300666d7"} Jul 27 19:27:08.659364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.659349 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-jfvdl" Jul 27 19:27:08.659364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.659376 2566 scope.go:117] "RemoveContainer" containerID="bc414c1931f8c13edb654aa7d8111413d6e2ca1adad9e23467271ce54b344218" Jul 27 19:27:08.691034 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.690997 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:27:08.697540 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:08.697509 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-jfvdl"] Jul 27 19:27:09.545867 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:09.545832 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d55f8dc-da6e-4770-9412-1e79ec7badd6" path="/var/lib/kubelet/pods/7d55f8dc-da6e-4770-9412-1e79ec7badd6/volumes" Jul 27 19:27:13.438250 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.438213 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cvxln"] Jul 27 19:27:13.438675 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.438662 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d55f8dc-da6e-4770-9412-1e79ec7badd6" containerName="limitador" Jul 27 19:27:13.438723 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.438676 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d55f8dc-da6e-4770-9412-1e79ec7badd6" containerName="limitador" Jul 27 19:27:13.438775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.438746 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d55f8dc-da6e-4770-9412-1e79ec7badd6" containerName="limitador" Jul 27 19:27:13.445089 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.445065 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.449490 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.449465 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 27 19:27:13.449809 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.449791 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 27 19:27:13.449892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.449792 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 27 19:27:13.449940 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.449820 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-hmwbm\"" Jul 27 19:27:13.456387 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.456363 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cvxln"] Jul 27 19:27:13.518931 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.518877 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chszv\" (UniqueName: \"kubernetes.io/projected/5a1018fe-445d-4793-bfb2-15565b6b0f77-kube-api-access-chszv\") pod \"postgres-69fb4594cf-cvxln\" (UID: \"5a1018fe-445d-4793-bfb2-15565b6b0f77\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.519175 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.518965 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a1018fe-445d-4793-bfb2-15565b6b0f77-data\") pod \"postgres-69fb4594cf-cvxln\" (UID: \"5a1018fe-445d-4793-bfb2-15565b6b0f77\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.619815 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.619772 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chszv\" (UniqueName: \"kubernetes.io/projected/5a1018fe-445d-4793-bfb2-15565b6b0f77-kube-api-access-chszv\") pod \"postgres-69fb4594cf-cvxln\" (UID: \"5a1018fe-445d-4793-bfb2-15565b6b0f77\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.620037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.619836 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a1018fe-445d-4793-bfb2-15565b6b0f77-data\") pod \"postgres-69fb4594cf-cvxln\" (UID: \"5a1018fe-445d-4793-bfb2-15565b6b0f77\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.620323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.620302 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/5a1018fe-445d-4793-bfb2-15565b6b0f77-data\") pod \"postgres-69fb4594cf-cvxln\" (UID: \"5a1018fe-445d-4793-bfb2-15565b6b0f77\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.631924 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.631898 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-chszv\" (UniqueName: \"kubernetes.io/projected/5a1018fe-445d-4793-bfb2-15565b6b0f77-kube-api-access-chszv\") pod \"postgres-69fb4594cf-cvxln\" (UID: \"5a1018fe-445d-4793-bfb2-15565b6b0f77\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.756537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.756447 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:13.902184 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:13.902158 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-cvxln"] Jul 27 19:27:13.904806 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:27:13.904750 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1018fe_445d_4793_bfb2_15565b6b0f77.slice/crio-5bab8f4b901997e0d2cff31365a0e7ee5404c2510ddb97f893eb8a9ab43ea113 WatchSource:0}: Error finding container 5bab8f4b901997e0d2cff31365a0e7ee5404c2510ddb97f893eb8a9ab43ea113: Status 404 returned error can't find the container with id 5bab8f4b901997e0d2cff31365a0e7ee5404c2510ddb97f893eb8a9ab43ea113 Jul 27 19:27:14.690807 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:14.690748 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" event={"ID":"5a1018fe-445d-4793-bfb2-15565b6b0f77","Type":"ContainerStarted","Data":"5bab8f4b901997e0d2cff31365a0e7ee5404c2510ddb97f893eb8a9ab43ea113"} Jul 27 19:27:19.722671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:19.722630 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" event={"ID":"5a1018fe-445d-4793-bfb2-15565b6b0f77","Type":"ContainerStarted","Data":"6b5c148315ae6786736b88a0d359b7a3945193c0af7b753e608ea19d376b278b"} Jul 27 19:27:19.723117 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:19.722783 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:19.740729 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:19.740668 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" podStartSLOduration=1.615786722 podStartE2EDuration="6.7406529s" totalImagesPullingTime="5.124866178s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:19.740365981 +0000 UTC m=+612.734862956" watchObservedRunningTime="2026-07-27 19:27:19.7406529 +0000 UTC m=+612.735149875" Jul 27 19:27:25.760896 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:25.760866 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-cvxln" Jul 27 19:27:33.453424 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.453334 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7c5888ff68-psj2z"] Jul 27 19:27:33.457030 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.457008 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.459867 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.459838 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lrg4n\"" Jul 27 19:27:33.460708 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.460685 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 19:27:33.460905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.460753 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 19:27:33.472374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.472339 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c5888ff68-psj2z"] Jul 27 19:27:33.617456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.617409 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e203765-5937-41ee-8692-60836dd71d0e-webhook-cert\") pod \"maas-controller-7c5888ff68-psj2z\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.617456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.617455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n7v5\" (UniqueName: \"kubernetes.io/projected/9e203765-5937-41ee-8692-60836dd71d0e-kube-api-access-5n7v5\") pod \"maas-controller-7c5888ff68-psj2z\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.655918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.655883 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f96b9564b-fq4c2"] Jul 27 19:27:33.659563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.659536 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.669896 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.669866 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-fq4c2"] Jul 27 19:27:33.718908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.718800 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e203765-5937-41ee-8692-60836dd71d0e-webhook-cert\") pod \"maas-controller-7c5888ff68-psj2z\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.718908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.718844 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n7v5\" (UniqueName: \"kubernetes.io/projected/9e203765-5937-41ee-8692-60836dd71d0e-kube-api-access-5n7v5\") pod \"maas-controller-7c5888ff68-psj2z\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.721376 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.721352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e203765-5937-41ee-8692-60836dd71d0e-webhook-cert\") pod \"maas-controller-7c5888ff68-psj2z\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.732491 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.732459 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n7v5\" (UniqueName: \"kubernetes.io/projected/9e203765-5937-41ee-8692-60836dd71d0e-kube-api-access-5n7v5\") pod \"maas-controller-7c5888ff68-psj2z\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.769315 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.769277 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:33.819662 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.819625 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-webhook-cert\") pod \"maas-controller-6f96b9564b-fq4c2\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.819662 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.819668 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfvq5\" (UniqueName: \"kubernetes.io/projected/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-kube-api-access-hfvq5\") pod \"maas-controller-6f96b9564b-fq4c2\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.920817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.920784 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-webhook-cert\") pod \"maas-controller-6f96b9564b-fq4c2\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.921018 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.920826 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfvq5\" (UniqueName: \"kubernetes.io/projected/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-kube-api-access-hfvq5\") pod \"maas-controller-6f96b9564b-fq4c2\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.922323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.922295 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7c5888ff68-psj2z"] Jul 27 19:27:33.923303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.923282 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-webhook-cert\") pod \"maas-controller-6f96b9564b-fq4c2\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.924110 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:27:33.924082 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e203765_5937_41ee_8692_60836dd71d0e.slice/crio-96f35b644fffd16a393f17fe3be7b762068df7e9bb7e53a2652cf4faa92ac09d WatchSource:0}: Error finding container 96f35b644fffd16a393f17fe3be7b762068df7e9bb7e53a2652cf4faa92ac09d: Status 404 returned error can't find the container with id 96f35b644fffd16a393f17fe3be7b762068df7e9bb7e53a2652cf4faa92ac09d Jul 27 19:27:33.925603 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.925586 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:27:33.930858 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.930837 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfvq5\" (UniqueName: \"kubernetes.io/projected/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-kube-api-access-hfvq5\") pod \"maas-controller-6f96b9564b-fq4c2\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:33.971665 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:33.971570 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:34.125955 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:34.125924 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-fq4c2"] Jul 27 19:27:34.128032 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:27:34.128001 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeed2cf5_7fcf_40aa_bcfb_c04e2759e8b3.slice/crio-5900b0eddb3abde166a099da2abee9dd5e454ca7c2c60b962a00b348ebb48bc3 WatchSource:0}: Error finding container 5900b0eddb3abde166a099da2abee9dd5e454ca7c2c60b962a00b348ebb48bc3: Status 404 returned error can't find the container with id 5900b0eddb3abde166a099da2abee9dd5e454ca7c2c60b962a00b348ebb48bc3 Jul 27 19:27:34.810330 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:34.810285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" event={"ID":"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3","Type":"ContainerStarted","Data":"5900b0eddb3abde166a099da2abee9dd5e454ca7c2c60b962a00b348ebb48bc3"} Jul 27 19:27:34.812347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:34.812314 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c5888ff68-psj2z" event={"ID":"9e203765-5937-41ee-8692-60836dd71d0e","Type":"ContainerStarted","Data":"96f35b644fffd16a393f17fe3be7b762068df7e9bb7e53a2652cf4faa92ac09d"} Jul 27 19:27:36.824705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:36.824602 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c5888ff68-psj2z" event={"ID":"9e203765-5937-41ee-8692-60836dd71d0e","Type":"ContainerStarted","Data":"e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349"} Jul 27 19:27:36.824705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:36.824696 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:36.826180 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:36.826147 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" event={"ID":"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3","Type":"ContainerStarted","Data":"711fd625511afbf6c97159a78823d19c995adac296c323ff670b34d82bacca33"} Jul 27 19:27:36.826309 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:36.826292 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:36.846079 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:36.846020 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7c5888ff68-psj2z" podStartSLOduration=1.232492863 podStartE2EDuration="3.846002416s" totalImagesPullingTime="2.613509553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:36.84387885 +0000 UTC m=+629.838375825" watchObservedRunningTime="2026-07-27 19:27:36.846002416 +0000 UTC m=+629.840499390" Jul 27 19:27:36.865554 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:36.865473 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" podStartSLOduration=1.463398757 podStartE2EDuration="3.865453377s" totalImagesPullingTime="2.40205462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:27:36.864406771 +0000 UTC m=+629.858903751" watchObservedRunningTime="2026-07-27 19:27:36.865453377 +0000 UTC m=+629.859950381" Jul 27 19:27:47.834234 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:47.834199 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:27:47.834775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:47.834431 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:47.922188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:47.922152 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c5888ff68-psj2z"] Jul 27 19:27:47.922400 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:47.922370 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7c5888ff68-psj2z" podUID="9e203765-5937-41ee-8692-60836dd71d0e" containerName="manager" containerID="cri-o://e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349" gracePeriod=30 Jul 27 19:27:48.183833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.183805 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:48.260836 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.260745 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e203765-5937-41ee-8692-60836dd71d0e-webhook-cert\") pod \"9e203765-5937-41ee-8692-60836dd71d0e\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " Jul 27 19:27:48.261022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.260878 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5n7v5\" (UniqueName: \"kubernetes.io/projected/9e203765-5937-41ee-8692-60836dd71d0e-kube-api-access-5n7v5\") pod \"9e203765-5937-41ee-8692-60836dd71d0e\" (UID: \"9e203765-5937-41ee-8692-60836dd71d0e\") " Jul 27 19:27:48.263087 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.263048 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e203765-5937-41ee-8692-60836dd71d0e-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "9e203765-5937-41ee-8692-60836dd71d0e" (UID: "9e203765-5937-41ee-8692-60836dd71d0e"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:27:48.263087 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.263066 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e203765-5937-41ee-8692-60836dd71d0e-kube-api-access-5n7v5" (OuterVolumeSpecName: "kube-api-access-5n7v5") pod "9e203765-5937-41ee-8692-60836dd71d0e" (UID: "9e203765-5937-41ee-8692-60836dd71d0e"). InnerVolumeSpecName "kube-api-access-5n7v5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:27:48.362480 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.362385 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n7v5\" (UniqueName: \"kubernetes.io/projected/9e203765-5937-41ee-8692-60836dd71d0e-kube-api-access-5n7v5\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:27:48.362480 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.362415 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9e203765-5937-41ee-8692-60836dd71d0e-webhook-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:27:48.893160 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.893130 2566 generic.go:356] "Generic (PLEG): container finished" podID="9e203765-5937-41ee-8692-60836dd71d0e" containerID="e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349" exitCode=0 Jul 27 19:27:48.893601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.893177 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7c5888ff68-psj2z" Jul 27 19:27:48.893601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.893222 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c5888ff68-psj2z" event={"ID":"9e203765-5937-41ee-8692-60836dd71d0e","Type":"ContainerDied","Data":"e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349"} Jul 27 19:27:48.893601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.893272 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7c5888ff68-psj2z" event={"ID":"9e203765-5937-41ee-8692-60836dd71d0e","Type":"ContainerDied","Data":"96f35b644fffd16a393f17fe3be7b762068df7e9bb7e53a2652cf4faa92ac09d"} Jul 27 19:27:48.893601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.893295 2566 scope.go:117] "RemoveContainer" containerID="e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349" Jul 27 19:27:48.902833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.902811 2566 scope.go:117] "RemoveContainer" containerID="e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349" Jul 27 19:27:48.903128 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:27:48.903110 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349\": container with ID starting with e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349 not found: ID does not exist" containerID="e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349" Jul 27 19:27:48.903186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.903140 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349"} err="failed to get container status \"e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349\": rpc error: code = NotFound desc = could not find container \"e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349\": container with ID starting with e90d8591a5b8772973ce93fb4841d64c27a35d957fed8ca579afb18be5735349 not found: ID does not exist" Jul 27 19:27:48.915132 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.915090 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7c5888ff68-psj2z"] Jul 27 19:27:48.920634 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:48.920603 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7c5888ff68-psj2z"] Jul 27 19:27:49.546833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:27:49.546800 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e203765-5937-41ee-8692-60836dd71d0e" path="/var/lib/kubelet/pods/9e203765-5937-41ee-8692-60836dd71d0e/volumes" Jul 27 19:28:09.334141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.334100 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq"] Jul 27 19:28:09.334553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.334539 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e203765-5937-41ee-8692-60836dd71d0e" containerName="manager" Jul 27 19:28:09.334553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.334551 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e203765-5937-41ee-8692-60836dd71d0e" containerName="manager" Jul 27 19:28:09.334642 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.334633 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e203765-5937-41ee-8692-60836dd71d0e" containerName="manager" Jul 27 19:28:09.341814 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.341745 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.345593 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.345551 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 27 19:28:09.345817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.345612 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dclfl\"" Jul 27 19:28:09.352911 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.352877 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq"] Jul 27 19:28:09.462152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.462119 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eaedbb-efe0-485a-9c29-70e68ff63b26-openshift-service-ca\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.462335 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.462175 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eaedbb-efe0-485a-9c29-70e68ff63b26-maas-api-tls\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.462335 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.462241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssrzl\" (UniqueName: \"kubernetes.io/projected/e2eaedbb-efe0-485a-9c29-70e68ff63b26-kube-api-access-ssrzl\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.563085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.563048 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eaedbb-efe0-485a-9c29-70e68ff63b26-openshift-service-ca\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.563273 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.563108 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eaedbb-efe0-485a-9c29-70e68ff63b26-maas-api-tls\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.563273 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.563168 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssrzl\" (UniqueName: \"kubernetes.io/projected/e2eaedbb-efe0-485a-9c29-70e68ff63b26-kube-api-access-ssrzl\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.563766 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.563733 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2eaedbb-efe0-485a-9c29-70e68ff63b26-openshift-service-ca\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.565812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.565777 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2eaedbb-efe0-485a-9c29-70e68ff63b26-maas-api-tls\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.578718 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.578679 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssrzl\" (UniqueName: \"kubernetes.io/projected/e2eaedbb-efe0-485a-9c29-70e68ff63b26-kube-api-access-ssrzl\") pod \"maas-api-76dd68f85-5fgqq\" (UID: \"e2eaedbb-efe0-485a-9c29-70e68ff63b26\") " pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.655642 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.655513 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:09.845345 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:09.845319 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq"] Jul 27 19:28:09.847889 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:28:09.847859 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2eaedbb_efe0_485a_9c29_70e68ff63b26.slice/crio-45b5ca7018ca906986cb1fb9b70ca28a5e3385589ee9492f5ac7dd53cde8f419 WatchSource:0}: Error finding container 45b5ca7018ca906986cb1fb9b70ca28a5e3385589ee9492f5ac7dd53cde8f419: Status 404 returned error can't find the container with id 45b5ca7018ca906986cb1fb9b70ca28a5e3385589ee9492f5ac7dd53cde8f419 Jul 27 19:28:10.005298 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.005265 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" event={"ID":"e2eaedbb-efe0-485a-9c29-70e68ff63b26","Type":"ContainerStarted","Data":"45b5ca7018ca906986cb1fb9b70ca28a5e3385589ee9492f5ac7dd53cde8f419"} Jul 27 19:28:10.071289 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.071252 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q"] Jul 27 19:28:10.077614 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.077589 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.081021 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.080992 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 27 19:28:10.097542 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.097506 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q"] Jul 27 19:28:10.170371 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.170328 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2gsf\" (UniqueName: \"kubernetes.io/projected/aeb8f814-39a2-4e03-9b40-5c6ab3981874-kube-api-access-g2gsf\") pod \"payload-pre-processing-687dbb6cf6-djt9q\" (UID: \"aeb8f814-39a2-4e03-9b40-5c6ab3981874\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.170604 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.170445 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8f814-39a2-4e03-9b40-5c6ab3981874-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-djt9q\" (UID: \"aeb8f814-39a2-4e03-9b40-5c6ab3981874\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.187829 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.187789 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-n54d2"] Jul 27 19:28:10.192380 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.192360 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.195727 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.195698 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-64mqw\"" Jul 27 19:28:10.217256 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.217210 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-n54d2"] Jul 27 19:28:10.271961 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.271858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8f814-39a2-4e03-9b40-5c6ab3981874-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-djt9q\" (UID: \"aeb8f814-39a2-4e03-9b40-5c6ab3981874\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.272120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.272068 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2gsf\" (UniqueName: \"kubernetes.io/projected/aeb8f814-39a2-4e03-9b40-5c6ab3981874-kube-api-access-g2gsf\") pod \"payload-pre-processing-687dbb6cf6-djt9q\" (UID: \"aeb8f814-39a2-4e03-9b40-5c6ab3981874\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.272581 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.272562 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb8f814-39a2-4e03-9b40-5c6ab3981874-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-djt9q\" (UID: \"aeb8f814-39a2-4e03-9b40-5c6ab3981874\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.293724 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.293688 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2gsf\" (UniqueName: \"kubernetes.io/projected/aeb8f814-39a2-4e03-9b40-5c6ab3981874-kube-api-access-g2gsf\") pod \"payload-pre-processing-687dbb6cf6-djt9q\" (UID: \"aeb8f814-39a2-4e03-9b40-5c6ab3981874\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.373032 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.372983 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcjmb\" (UniqueName: \"kubernetes.io/projected/21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e-kube-api-access-rcjmb\") pod \"payload-processing-5cc6cd444b-n54d2\" (UID: \"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.373435 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.373039 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-n54d2\" (UID: \"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.396084 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.396042 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:10.474503 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.474465 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcjmb\" (UniqueName: \"kubernetes.io/projected/21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e-kube-api-access-rcjmb\") pod \"payload-processing-5cc6cd444b-n54d2\" (UID: \"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.474503 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.474510 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-n54d2\" (UID: \"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.475258 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.475234 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-n54d2\" (UID: \"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.487035 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.486993 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcjmb\" (UniqueName: \"kubernetes.io/projected/21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e-kube-api-access-rcjmb\") pod \"payload-processing-5cc6cd444b-n54d2\" (UID: \"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e\") " pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.503539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.503502 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:10.554215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.554176 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q"] Jul 27 19:28:10.690037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:10.690007 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-n54d2"] Jul 27 19:28:10.692811 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:28:10.692755 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d4d1a8_cef0_43d6_ac2b_7be3a3f7694e.slice/crio-2243f83c50407420203f9b7c46da1a9b8b492a17ef62060ab5b3c8eac91557d9 WatchSource:0}: Error finding container 2243f83c50407420203f9b7c46da1a9b8b492a17ef62060ab5b3c8eac91557d9: Status 404 returned error can't find the container with id 2243f83c50407420203f9b7c46da1a9b8b492a17ef62060ab5b3c8eac91557d9 Jul 27 19:28:11.035526 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:11.035408 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" event={"ID":"aeb8f814-39a2-4e03-9b40-5c6ab3981874","Type":"ContainerStarted","Data":"dfa2d8132704f006bc387fa71667e8405ae6298ffbfc1a205c2ad16a591dfcf3"} Jul 27 19:28:11.041829 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:11.041668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" event={"ID":"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e","Type":"ContainerStarted","Data":"2243f83c50407420203f9b7c46da1a9b8b492a17ef62060ab5b3c8eac91557d9"} Jul 27 19:28:13.063622 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:13.063591 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" event={"ID":"e2eaedbb-efe0-485a-9c29-70e68ff63b26","Type":"ContainerStarted","Data":"f86e072da6ff380961f08d934a1ab0c6ae4413cbbddcf9a284c7522dfa1d8208"} Jul 27 19:28:13.064152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:13.064134 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:13.084239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:13.084193 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" podStartSLOduration=1.986875236 podStartE2EDuration="4.084177923s" totalImagesPullingTime="2.097302687s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:28:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:13.083179676 +0000 UTC m=+666.077676652" watchObservedRunningTime="2026-07-27 19:28:13.084177923 +0000 UTC m=+666.078674907" Jul 27 19:28:14.070619 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:14.070584 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" event={"ID":"21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e","Type":"ContainerStarted","Data":"be0506cc5e733718fcd3b55d8d56b7d7358989b967927bab08be82ebc7d96c0e"} Jul 27 19:28:14.071116 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:14.070653 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:14.072328 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:14.072295 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" event={"ID":"aeb8f814-39a2-4e03-9b40-5c6ab3981874","Type":"ContainerStarted","Data":"feae21a338541d0e974e58ad6c5692c782a4f56eaf04ca33cbe2534c814039d7"} Jul 27 19:28:14.093938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:14.093875 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" podStartSLOduration=1.656144309 podStartE2EDuration="4.093851242s" totalImagesPullingTime="2.437706933s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:14.093259871 +0000 UTC m=+667.087756850" watchObservedRunningTime="2026-07-27 19:28:14.093851242 +0000 UTC m=+667.088348218" Jul 27 19:28:14.120032 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:14.119961 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" podStartSLOduration=1.6750257899999998 podStartE2EDuration="4.119938005s" totalImagesPullingTime="2.444912215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:28:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:14.11664686 +0000 UTC m=+667.111143838" watchObservedRunningTime="2026-07-27 19:28:14.119938005 +0000 UTC m=+667.114434982" Jul 27 19:28:15.078893 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:15.078858 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:19.100979 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:19.100946 2566 generic.go:356] "Generic (PLEG): container finished" podID="61b8a2c6-049e-4451-9c7d-7a0670d98ba1" containerID="8b91a177e6c751ab83179ce409b85d4340fe56376faf0d7604d81a19409fa014" exitCode=0 Jul 27 19:28:19.100979 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:19.100985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" event={"ID":"61b8a2c6-049e-4451-9c7d-7a0670d98ba1","Type":"ContainerDied","Data":"8b91a177e6c751ab83179ce409b85d4340fe56376faf0d7604d81a19409fa014"} Jul 27 19:28:19.101530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:19.101385 2566 scope.go:117] "RemoveContainer" containerID="8b91a177e6c751ab83179ce409b85d4340fe56376faf0d7604d81a19409fa014" Jul 27 19:28:19.102527 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:19.102495 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:28:19.102584 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:19.102567 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:28:19.102625 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:19.102609 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:28:20.085135 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:20.085105 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-76dd68f85-5fgqq" Jul 27 19:28:20.108308 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:20.108274 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" event={"ID":"61b8a2c6-049e-4451-9c7d-7a0670d98ba1","Type":"ContainerStarted","Data":"7102c1c3802b9f82730db2b9e659a96765d634f4b985e3f586e552eea3453789"} Jul 27 19:28:20.180152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:20.180121 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:28:20.181456 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:20.181429 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tqprs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 19:28:20.181574 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:20.181479 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" podUID="61b8a2c6-049e-4451-9c7d-7a0670d98ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 19:28:21.180486 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:21.180443 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tqprs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 19:28:21.180936 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:21.180530 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" podUID="61b8a2c6-049e-4451-9c7d-7a0670d98ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 19:28:21.565457 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:21.565354 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:28:22.180021 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:22.179985 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tqprs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 19:28:22.180216 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:22.180045 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" podUID="61b8a2c6-049e-4451-9c7d-7a0670d98ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 19:28:23.180549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:23.180505 2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-5ccb968c66-tqprs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 27 19:28:23.181099 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:23.180584 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" podUID="61b8a2c6-049e-4451-9c7d-7a0670d98ba1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 27 19:28:24.183943 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:24.183909 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:28:24.185280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:24.185256 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5ccb968c66-tqprs" Jul 27 19:28:25.081705 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:25.081673 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-n54d2" Jul 27 19:28:26.086149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.086116 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-djt9q" Jul 27 19:28:26.704751 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.704717 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96"] Jul 27 19:28:26.709745 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.709724 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.714777 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.714724 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zs4xt\"" Jul 27 19:28:26.715167 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.715152 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 27 19:28:26.715512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.715493 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 27 19:28:26.715564 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.715529 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 27 19:28:26.722152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.722125 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96"] Jul 27 19:28:26.742001 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.741956 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.742186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.742017 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.742186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.742067 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.742186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.742142 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6p47\" (UniqueName: \"kubernetes.io/projected/7689531c-be6e-428c-afd0-7d81695ebaa7-kube-api-access-j6p47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.742324 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.742247 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7689531c-be6e-428c-afd0-7d81695ebaa7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.742324 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.742291 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.843454 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843410 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.843655 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843466 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6p47\" (UniqueName: \"kubernetes.io/projected/7689531c-be6e-428c-afd0-7d81695ebaa7-kube-api-access-j6p47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.843655 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843528 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7689531c-be6e-428c-afd0-7d81695ebaa7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.843655 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843552 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.843655 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843578 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.843655 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843623 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.844008 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.843982 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.844074 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.844051 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.844192 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.844173 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.846064 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.846043 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7689531c-be6e-428c-afd0-7d81695ebaa7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.846178 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.846117 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7689531c-be6e-428c-afd0-7d81695ebaa7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:26.852848 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:26.852809 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6p47\" (UniqueName: \"kubernetes.io/projected/7689531c-be6e-428c-afd0-7d81695ebaa7-kube-api-access-j6p47\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96\" (UID: \"7689531c-be6e-428c-afd0-7d81695ebaa7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:27.021096 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:27.021006 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:27.171061 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:27.171034 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96"] Jul 27 19:28:27.173456 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:28:27.173427 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7689531c_be6e_428c_afd0_7d81695ebaa7.slice/crio-28f42d2ef2444d310f6885b04b41d544dba688cc6d29c30a6e4f075eebdff0c0 WatchSource:0}: Error finding container 28f42d2ef2444d310f6885b04b41d544dba688cc6d29c30a6e4f075eebdff0c0: Status 404 returned error can't find the container with id 28f42d2ef2444d310f6885b04b41d544dba688cc6d29c30a6e4f075eebdff0c0 Jul 27 19:28:28.169511 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:28.169477 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" event={"ID":"7689531c-be6e-428c-afd0-7d81695ebaa7","Type":"ContainerStarted","Data":"28f42d2ef2444d310f6885b04b41d544dba688cc6d29c30a6e4f075eebdff0c0"} Jul 27 19:28:29.176865 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:29.176814 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" event={"ID":"7689531c-be6e-428c-afd0-7d81695ebaa7","Type":"ContainerStarted","Data":"aed304925b0157c4476637a3bed6b77e494d8d1c0e4136a12fab6990bc03a97a"} Jul 27 19:28:29.225276 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:29.225223 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" podStartSLOduration=2.212130939 podStartE2EDuration="3.225205894s" totalImagesPullingTime="1.013074955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:28:29.223811887 +0000 UTC m=+682.218308865" watchObservedRunningTime="2026-07-27 19:28:29.225205894 +0000 UTC m=+682.219702868" Jul 27 19:28:37.021645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:37.021601 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:37.022098 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:37.021675 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:37.034971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:37.034939 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:28:37.239699 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:28:37.239670 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96" Jul 27 19:30:00.153913 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.153876 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv"] Jul 27 19:30:00.157838 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.157814 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" Jul 27 19:30:00.160394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.160366 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:30:00.166471 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.165858 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv"] Jul 27 19:30:00.171008 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.170980 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" Jul 27 19:30:00.530172 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.530140 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv"] Jul 27 19:30:00.532511 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:30:00.532479 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e808457_6f72_4140_acc5_f5ceea91d619.slice/crio-01fb974d0d53d61ad1bfe1fd40016405175624d14da389d7e235fb2eee18bc73 WatchSource:0}: Error finding container 01fb974d0d53d61ad1bfe1fd40016405175624d14da389d7e235fb2eee18bc73: Status 404 returned error can't find the container with id 01fb974d0d53d61ad1bfe1fd40016405175624d14da389d7e235fb2eee18bc73 Jul 27 19:30:00.691996 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:00.691957 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" event={"ID":"4e808457-6f72-4140-acc5-f5ceea91d619","Type":"ContainerStarted","Data":"01fb974d0d53d61ad1bfe1fd40016405175624d14da389d7e235fb2eee18bc73"} Jul 27 19:30:03.713200 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:03.713155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" event={"ID":"4e808457-6f72-4140-acc5-f5ceea91d619","Type":"ContainerStarted","Data":"d5692de5a23c025337e43d144e7127c228485e9c7005f186791c6e40ca8512ef"} Jul 27 19:30:03.730738 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:03.730681 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" podStartSLOduration=0.968962487 podStartE2EDuration="3.730663575s" totalImagesPullingTime="2.761701088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:30:03.729200644 +0000 UTC m=+776.723697620" watchObservedRunningTime="2026-07-27 19:30:03.730663575 +0000 UTC m=+776.725160550" Jul 27 19:30:04.719937 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:04.719900 2566 generic.go:356] "Generic (PLEG): container finished" podID="4e808457-6f72-4140-acc5-f5ceea91d619" containerID="d5692de5a23c025337e43d144e7127c228485e9c7005f186791c6e40ca8512ef" exitCode=0 Jul 27 19:30:04.720405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:04.719988 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" event={"ID":"4e808457-6f72-4140-acc5-f5ceea91d619","Type":"ContainerDied","Data":"d5692de5a23c025337e43d144e7127c228485e9c7005f186791c6e40ca8512ef"} Jul 27 19:30:05.881358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:05.881330 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" Jul 27 19:30:06.733696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:06.733668 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" Jul 27 19:30:06.733949 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:06.733701 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753010-qwqhv" event={"ID":"4e808457-6f72-4140-acc5-f5ceea91d619","Type":"ContainerDied","Data":"01fb974d0d53d61ad1bfe1fd40016405175624d14da389d7e235fb2eee18bc73"} Jul 27 19:30:06.733949 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:30:06.733746 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01fb974d0d53d61ad1bfe1fd40016405175624d14da389d7e235fb2eee18bc73" Jul 27 19:31:30.037365 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.037324 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-fq4c2"] Jul 27 19:31:30.037995 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.037629 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" podUID="deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" containerName="manager" containerID="cri-o://711fd625511afbf6c97159a78823d19c995adac296c323ff670b34d82bacca33" gracePeriod=30 Jul 27 19:31:30.203941 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.203906 2566 generic.go:356] "Generic (PLEG): container finished" podID="deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" containerID="711fd625511afbf6c97159a78823d19c995adac296c323ff670b34d82bacca33" exitCode=0 Jul 27 19:31:30.204115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.203975 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" event={"ID":"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3","Type":"ContainerDied","Data":"711fd625511afbf6c97159a78823d19c995adac296c323ff670b34d82bacca33"} Jul 27 19:31:30.303054 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.302976 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:31:30.372028 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.371993 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-webhook-cert\") pod \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " Jul 27 19:31:30.372194 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.372066 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfvq5\" (UniqueName: \"kubernetes.io/projected/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-kube-api-access-hfvq5\") pod \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\" (UID: \"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3\") " Jul 27 19:31:30.374173 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.374137 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" (UID: "deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:31:30.374326 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.374249 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-kube-api-access-hfvq5" (OuterVolumeSpecName: "kube-api-access-hfvq5") pod "deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" (UID: "deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3"). InnerVolumeSpecName "kube-api-access-hfvq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:31:30.473896 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.473859 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-webhook-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:31:30.473896 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:30.473892 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfvq5\" (UniqueName: \"kubernetes.io/projected/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3-kube-api-access-hfvq5\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:31:31.211215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:31.211183 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" Jul 27 19:31:31.211215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:31.211209 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-fq4c2" event={"ID":"deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3","Type":"ContainerDied","Data":"5900b0eddb3abde166a099da2abee9dd5e454ca7c2c60b962a00b348ebb48bc3"} Jul 27 19:31:31.211812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:31.211265 2566 scope.go:117] "RemoveContainer" containerID="711fd625511afbf6c97159a78823d19c995adac296c323ff670b34d82bacca33" Jul 27 19:31:31.239793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:31.239746 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-fq4c2"] Jul 27 19:31:31.243605 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:31.243566 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-fq4c2"] Jul 27 19:31:31.546260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:31:31.546176 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" path="/var/lib/kubelet/pods/deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3/volumes" Jul 27 19:32:07.526081 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:07.526001 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:32:07.528643 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:07.528614 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:32:32.034311 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034270 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f96b9564b-v54dp"] Jul 27 19:32:32.034778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034719 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e808457-6f72-4140-acc5-f5ceea91d619" containerName="cleanup" Jul 27 19:32:32.034778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034730 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e808457-6f72-4140-acc5-f5ceea91d619" containerName="cleanup" Jul 27 19:32:32.034778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034742 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" containerName="manager" Jul 27 19:32:32.034778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034747 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" containerName="manager" Jul 27 19:32:32.034926 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034860 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e808457-6f72-4140-acc5-f5ceea91d619" containerName="cleanup" Jul 27 19:32:32.034926 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.034874 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="deed2cf5-7fcf-40aa-bcfb-c04e2759e8b3" containerName="manager" Jul 27 19:32:32.038181 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.038165 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.041452 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.041424 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 19:32:32.041602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.041458 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lrg4n\"" Jul 27 19:32:32.041602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.041426 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 19:32:32.046623 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.046595 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-v54dp"] Jul 27 19:32:32.092640 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.092595 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95590836-ca07-481a-8a05-325529d31a1f-webhook-cert\") pod \"maas-controller-6f96b9564b-v54dp\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.092859 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.092690 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmbd\" (UniqueName: \"kubernetes.io/projected/95590836-ca07-481a-8a05-325529d31a1f-kube-api-access-5zmbd\") pod \"maas-controller-6f96b9564b-v54dp\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.193803 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.193748 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmbd\" (UniqueName: \"kubernetes.io/projected/95590836-ca07-481a-8a05-325529d31a1f-kube-api-access-5zmbd\") pod \"maas-controller-6f96b9564b-v54dp\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.193976 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.193875 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95590836-ca07-481a-8a05-325529d31a1f-webhook-cert\") pod \"maas-controller-6f96b9564b-v54dp\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.196379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.196352 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95590836-ca07-481a-8a05-325529d31a1f-webhook-cert\") pod \"maas-controller-6f96b9564b-v54dp\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.203517 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.203488 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmbd\" (UniqueName: \"kubernetes.io/projected/95590836-ca07-481a-8a05-325529d31a1f-kube-api-access-5zmbd\") pod \"maas-controller-6f96b9564b-v54dp\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.350204 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.350110 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:32.505543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.505511 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-v54dp"] Jul 27 19:32:32.508288 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:32:32.508251 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95590836_ca07_481a_8a05_325529d31a1f.slice/crio-7b3e15dbefce5b420d9107239323f9c284b905fb35141974eb6d52b996e5ce71 WatchSource:0}: Error finding container 7b3e15dbefce5b420d9107239323f9c284b905fb35141974eb6d52b996e5ce71: Status 404 returned error can't find the container with id 7b3e15dbefce5b420d9107239323f9c284b905fb35141974eb6d52b996e5ce71 Jul 27 19:32:32.551121 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:32.551088 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-v54dp" event={"ID":"95590836-ca07-481a-8a05-325529d31a1f","Type":"ContainerStarted","Data":"7b3e15dbefce5b420d9107239323f9c284b905fb35141974eb6d52b996e5ce71"} Jul 27 19:32:33.558327 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:33.558285 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-v54dp" event={"ID":"95590836-ca07-481a-8a05-325529d31a1f","Type":"ContainerStarted","Data":"24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4"} Jul 27 19:32:33.558860 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:33.558460 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:33.581750 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:33.581689 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f96b9564b-v54dp" podStartSLOduration=0.939456375 podStartE2EDuration="1.581672375s" totalImagesPullingTime="642.216ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:32:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:32:33.578229386 +0000 UTC m=+926.572726360" watchObservedRunningTime="2026-07-27 19:32:33.581672375 +0000 UTC m=+926.576169351" Jul 27 19:32:44.567695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:44.567661 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:49.192858 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.192811 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-v54dp"] Jul 27 19:32:49.193338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.193067 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-6f96b9564b-v54dp" podUID="95590836-ca07-481a-8a05-325529d31a1f" containerName="manager" containerID="cri-o://24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4" gracePeriod=30 Jul 27 19:32:49.460141 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.460073 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:49.574403 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.574364 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zmbd\" (UniqueName: \"kubernetes.io/projected/95590836-ca07-481a-8a05-325529d31a1f-kube-api-access-5zmbd\") pod \"95590836-ca07-481a-8a05-325529d31a1f\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " Jul 27 19:32:49.574604 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.574415 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95590836-ca07-481a-8a05-325529d31a1f-webhook-cert\") pod \"95590836-ca07-481a-8a05-325529d31a1f\" (UID: \"95590836-ca07-481a-8a05-325529d31a1f\") " Jul 27 19:32:49.576568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.576542 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95590836-ca07-481a-8a05-325529d31a1f-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "95590836-ca07-481a-8a05-325529d31a1f" (UID: "95590836-ca07-481a-8a05-325529d31a1f"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 19:32:49.576658 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.576543 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95590836-ca07-481a-8a05-325529d31a1f-kube-api-access-5zmbd" (OuterVolumeSpecName: "kube-api-access-5zmbd") pod "95590836-ca07-481a-8a05-325529d31a1f" (UID: "95590836-ca07-481a-8a05-325529d31a1f"). InnerVolumeSpecName "kube-api-access-5zmbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:32:49.650370 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.650330 2566 generic.go:356] "Generic (PLEG): container finished" podID="95590836-ca07-481a-8a05-325529d31a1f" containerID="24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4" exitCode=0 Jul 27 19:32:49.650549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.650386 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-v54dp" Jul 27 19:32:49.650549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.650415 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-v54dp" event={"ID":"95590836-ca07-481a-8a05-325529d31a1f","Type":"ContainerDied","Data":"24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4"} Jul 27 19:32:49.650549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.650469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-v54dp" event={"ID":"95590836-ca07-481a-8a05-325529d31a1f","Type":"ContainerDied","Data":"7b3e15dbefce5b420d9107239323f9c284b905fb35141974eb6d52b996e5ce71"} Jul 27 19:32:49.650549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.650491 2566 scope.go:117] "RemoveContainer" containerID="24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4" Jul 27 19:32:49.660278 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.660255 2566 scope.go:117] "RemoveContainer" containerID="24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4" Jul 27 19:32:49.660625 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:32:49.660602 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4\": container with ID starting with 24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4 not found: ID does not exist" containerID="24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4" Jul 27 19:32:49.660687 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.660635 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4"} err="failed to get container status \"24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4\": rpc error: code = NotFound desc = could not find container \"24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4\": container with ID starting with 24e733ac3f2eac92a3ab0d14e94fde6c94ce64524c1288b647191894a17ef5e4 not found: ID does not exist" Jul 27 19:32:49.673392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.673356 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-v54dp"] Jul 27 19:32:49.675153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.675106 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmbd\" (UniqueName: \"kubernetes.io/projected/95590836-ca07-481a-8a05-325529d31a1f-kube-api-access-5zmbd\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:32:49.675153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.675133 2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/95590836-ca07-481a-8a05-325529d31a1f-webhook-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:32:49.676892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:49.676871 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-v54dp"] Jul 27 19:32:50.524303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.524264 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6f96b9564b-njdhs"] Jul 27 19:32:50.525071 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.524729 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95590836-ca07-481a-8a05-325529d31a1f" containerName="manager" Jul 27 19:32:50.525071 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.524742 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="95590836-ca07-481a-8a05-325529d31a1f" containerName="manager" Jul 27 19:32:50.525071 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.524854 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="95590836-ca07-481a-8a05-325529d31a1f" containerName="manager" Jul 27 19:32:50.535603 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.535567 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.537815 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.537783 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-njdhs"] Jul 27 19:32:50.539138 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.539109 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 27 19:32:50.539263 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.539164 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 27 19:32:50.539307 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.539118 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-lrg4n\"" Jul 27 19:32:50.687072 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.687038 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbl7d\" (UniqueName: \"kubernetes.io/projected/0749ba4f-015e-4d0f-bc0d-dd26067f3140-kube-api-access-vbl7d\") pod \"maas-controller-6f96b9564b-njdhs\" (UID: \"0749ba4f-015e-4d0f-bc0d-dd26067f3140\") " pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.687272 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.687172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0749ba4f-015e-4d0f-bc0d-dd26067f3140-webhook-cert\") pod \"maas-controller-6f96b9564b-njdhs\" (UID: \"0749ba4f-015e-4d0f-bc0d-dd26067f3140\") " pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.788753 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.788650 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0749ba4f-015e-4d0f-bc0d-dd26067f3140-webhook-cert\") pod \"maas-controller-6f96b9564b-njdhs\" (UID: \"0749ba4f-015e-4d0f-bc0d-dd26067f3140\") " pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.788753 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.788723 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbl7d\" (UniqueName: \"kubernetes.io/projected/0749ba4f-015e-4d0f-bc0d-dd26067f3140-kube-api-access-vbl7d\") pod \"maas-controller-6f96b9564b-njdhs\" (UID: \"0749ba4f-015e-4d0f-bc0d-dd26067f3140\") " pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.791320 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.791289 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0749ba4f-015e-4d0f-bc0d-dd26067f3140-webhook-cert\") pod \"maas-controller-6f96b9564b-njdhs\" (UID: \"0749ba4f-015e-4d0f-bc0d-dd26067f3140\") " pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.798237 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.798207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbl7d\" (UniqueName: \"kubernetes.io/projected/0749ba4f-015e-4d0f-bc0d-dd26067f3140-kube-api-access-vbl7d\") pod \"maas-controller-6f96b9564b-njdhs\" (UID: \"0749ba4f-015e-4d0f-bc0d-dd26067f3140\") " pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:50.849292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:50.849257 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:51.001161 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.001133 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6f96b9564b-njdhs"] Jul 27 19:32:51.003470 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:32:51.003438 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0749ba4f_015e_4d0f_bc0d_dd26067f3140.slice/crio-fd97769cf78b00e50fbf85b94004458433f48c9561e0edf4a621bfc617d4b6e3 WatchSource:0}: Error finding container fd97769cf78b00e50fbf85b94004458433f48c9561e0edf4a621bfc617d4b6e3: Status 404 returned error can't find the container with id fd97769cf78b00e50fbf85b94004458433f48c9561e0edf4a621bfc617d4b6e3 Jul 27 19:32:51.005092 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.005070 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 19:32:51.548539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.548502 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95590836-ca07-481a-8a05-325529d31a1f" path="/var/lib/kubelet/pods/95590836-ca07-481a-8a05-325529d31a1f/volumes" Jul 27 19:32:51.667560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.667468 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-njdhs" event={"ID":"0749ba4f-015e-4d0f-bc0d-dd26067f3140","Type":"ContainerStarted","Data":"e1743d4b1b47733fd0d6428e3ded8475f8aeefe0e3a12b795f98bb132b4d6ef5"} Jul 27 19:32:51.667560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.667521 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6f96b9564b-njdhs" event={"ID":"0749ba4f-015e-4d0f-bc0d-dd26067f3140","Type":"ContainerStarted","Data":"fd97769cf78b00e50fbf85b94004458433f48c9561e0edf4a621bfc617d4b6e3"} Jul 27 19:32:51.667560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.667550 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:32:51.687453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:32:51.687392 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6f96b9564b-njdhs" podStartSLOduration=1.280545207 podStartE2EDuration="1.687377276s" totalImagesPullingTime="406.832069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:32:51.685575938 +0000 UTC m=+944.680072914" watchObservedRunningTime="2026-07-27 19:32:51.687377276 +0000 UTC m=+944.681874292" Jul 27 19:33:02.675546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:33:02.675513 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6f96b9564b-njdhs" Jul 27 19:36:40.239606 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.239523 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7cbccf9876-mtq5l"] Jul 27 19:36:40.243466 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.243443 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.246481 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.246453 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 27 19:36:40.247788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.247520 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 27 19:36:40.247788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.247556 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 27 19:36:40.247788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.247738 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 27 19:36:40.248049 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.247910 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 27 19:36:40.250317 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.250294 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 27 19:36:40.251074 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.251053 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 27 19:36:40.251202 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.251115 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-bhwfx\"" Jul 27 19:36:40.253824 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.253803 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 27 19:36:40.276653 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.276612 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cbccf9876-mtq5l"] Jul 27 19:36:40.284170 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284129 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90819a44-c71e-4337-89cf-ee6993332b65-console-serving-cert\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.284339 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284245 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-oauth-serving-cert\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.284339 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284320 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c54x\" (UniqueName: \"kubernetes.io/projected/90819a44-c71e-4337-89cf-ee6993332b65-kube-api-access-9c54x\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.284423 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284400 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-console-config\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.284462 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284431 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-trusted-ca-bundle\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.284512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284491 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90819a44-c71e-4337-89cf-ee6993332b65-console-oauth-config\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.284557 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.284514 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-service-ca\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385023 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90819a44-c71e-4337-89cf-ee6993332b65-console-oauth-config\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385067 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-service-ca\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385093 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90819a44-c71e-4337-89cf-ee6993332b65-console-serving-cert\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385152 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-oauth-serving-cert\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385232 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c54x\" (UniqueName: \"kubernetes.io/projected/90819a44-c71e-4337-89cf-ee6993332b65-kube-api-access-9c54x\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385286 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-console-config\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.385323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-trusted-ca-bundle\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.386005 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385954 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-service-ca\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.386005 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.385993 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-oauth-serving-cert\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.386319 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.386298 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-console-config\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.386389 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.386297 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/90819a44-c71e-4337-89cf-ee6993332b65-trusted-ca-bundle\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.387740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.387718 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/90819a44-c71e-4337-89cf-ee6993332b65-console-serving-cert\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.387866 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.387723 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/90819a44-c71e-4337-89cf-ee6993332b65-console-oauth-config\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.395529 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.395492 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c54x\" (UniqueName: \"kubernetes.io/projected/90819a44-c71e-4337-89cf-ee6993332b65-kube-api-access-9c54x\") pod \"console-7cbccf9876-mtq5l\" (UID: \"90819a44-c71e-4337-89cf-ee6993332b65\") " pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.555843 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.555681 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:40.718057 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.718029 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7cbccf9876-mtq5l"] Jul 27 19:36:40.720531 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:36:40.720501 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90819a44_c71e_4337_89cf_ee6993332b65.slice/crio-9cb04bc57a4842140764550c12c932e168ba8b56d3473bdabf76c6e29819568d WatchSource:0}: Error finding container 9cb04bc57a4842140764550c12c932e168ba8b56d3473bdabf76c6e29819568d: Status 404 returned error can't find the container with id 9cb04bc57a4842140764550c12c932e168ba8b56d3473bdabf76c6e29819568d Jul 27 19:36:40.962076 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.962039 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cbccf9876-mtq5l" event={"ID":"90819a44-c71e-4337-89cf-ee6993332b65","Type":"ContainerStarted","Data":"2d23cc57d389012ea1c9cd70905c627e83a8c8a95543e99053f4b9b883899d0c"} Jul 27 19:36:40.962297 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.962093 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7cbccf9876-mtq5l" event={"ID":"90819a44-c71e-4337-89cf-ee6993332b65","Type":"ContainerStarted","Data":"9cb04bc57a4842140764550c12c932e168ba8b56d3473bdabf76c6e29819568d"} Jul 27 19:36:40.979773 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:40.979697 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7cbccf9876-mtq5l" podStartSLOduration=0.979673725 podStartE2EDuration="979.673725ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:36:40.979362497 +0000 UTC m=+1173.973859496" watchObservedRunningTime="2026-07-27 19:36:40.979673725 +0000 UTC m=+1173.974170700" Jul 27 19:36:50.556397 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:50.556281 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:50.556397 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:50.556383 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:50.561488 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:50.561465 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:36:51.024388 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:36:51.024356 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7cbccf9876-mtq5l" Jul 27 19:37:07.569067 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:07.569041 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:37:07.569518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:07.569080 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:37:33.297131 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:33.297091 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk"] Jul 27 19:37:33.297729 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:33.297292 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" podUID="67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" containerName="manager" containerID="cri-o://548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f" gracePeriod=10 Jul 27 19:37:36.868816 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:36.868712 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:37:36.961538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:36.961491 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-extensions-socket-volume\") pod \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " Jul 27 19:37:36.961731 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:36.961619 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nhglf\" (UniqueName: \"kubernetes.io/projected/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-kube-api-access-nhglf\") pod \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\" (UID: \"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e\") " Jul 27 19:37:36.962022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:36.961996 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" (UID: "67b0e406-5f19-4aea-a13c-c6a5bcb0b83e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:37:36.963825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:36.963785 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-kube-api-access-nhglf" (OuterVolumeSpecName: "kube-api-access-nhglf") pod "67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" (UID: "67b0e406-5f19-4aea-a13c-c6a5bcb0b83e"). InnerVolumeSpecName "kube-api-access-nhglf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:37:37.063093 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.063048 2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-extensions-socket-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:37:37.063093 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.063083 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhglf\" (UniqueName: \"kubernetes.io/projected/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e-kube-api-access-nhglf\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:37:37.285793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.285720 2566 generic.go:356] "Generic (PLEG): container finished" podID="67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" containerID="548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f" exitCode=0 Jul 27 19:37:37.285793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.285787 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" event={"ID":"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e","Type":"ContainerDied","Data":"548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f"} Jul 27 19:37:37.286061 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.285798 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" Jul 27 19:37:37.286061 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.285876 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk" event={"ID":"67b0e406-5f19-4aea-a13c-c6a5bcb0b83e","Type":"ContainerDied","Data":"22867caf18e2330a5c3df95252efcd37f3d36193cd3e0ba32ac4980ea11dfa30"} Jul 27 19:37:37.286061 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.285909 2566 scope.go:117] "RemoveContainer" containerID="548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f" Jul 27 19:37:37.297171 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.297149 2566 scope.go:117] "RemoveContainer" containerID="548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f" Jul 27 19:37:37.299602 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:37:37.297919 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f\": container with ID starting with 548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f not found: ID does not exist" containerID="548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f" Jul 27 19:37:37.299602 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.297965 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f"} err="failed to get container status \"548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f\": rpc error: code = NotFound desc = could not find container \"548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f\": container with ID starting with 548c37bbfb8d17235007ee96dc8ae61b62ae31f541303beab46136e82e2a522f not found: ID does not exist" Jul 27 19:37:37.311378 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.311345 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk"] Jul 27 19:37:37.313873 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.313852 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-47vpk"] Jul 27 19:37:37.546567 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:37:37.546483 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" path="/var/lib/kubelet/pods/67b0e406-5f19-4aea-a13c-c6a5bcb0b83e/volumes" Jul 27 19:38:34.442280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.442239 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b"] Jul 27 19:38:34.442838 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.442739 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" containerName="manager" Jul 27 19:38:34.442838 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.442753 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" containerName="manager" Jul 27 19:38:34.442938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.442847 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b0e406-5f19-4aea-a13c-c6a5bcb0b83e" containerName="manager" Jul 27 19:38:34.447374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.447351 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.450825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.450793 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-s8tnh\"" Jul 27 19:38:34.465403 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.465366 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b"] Jul 27 19:38:34.529249 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.529203 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db6abb4d-8ae3-4324-bc0c-f542c6d4d743-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z299b\" (UID: \"db6abb4d-8ae3-4324-bc0c-f542c6d4d743\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.529442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.529279 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpmzw\" (UniqueName: \"kubernetes.io/projected/db6abb4d-8ae3-4324-bc0c-f542c6d4d743-kube-api-access-tpmzw\") pod \"kuadrant-operator-controller-manager-86878c769f-z299b\" (UID: \"db6abb4d-8ae3-4324-bc0c-f542c6d4d743\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.630429 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.630389 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db6abb4d-8ae3-4324-bc0c-f542c6d4d743-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z299b\" (UID: \"db6abb4d-8ae3-4324-bc0c-f542c6d4d743\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.630613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.630471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpmzw\" (UniqueName: \"kubernetes.io/projected/db6abb4d-8ae3-4324-bc0c-f542c6d4d743-kube-api-access-tpmzw\") pod \"kuadrant-operator-controller-manager-86878c769f-z299b\" (UID: \"db6abb4d-8ae3-4324-bc0c-f542c6d4d743\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.630846 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.630826 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/db6abb4d-8ae3-4324-bc0c-f542c6d4d743-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z299b\" (UID: \"db6abb4d-8ae3-4324-bc0c-f542c6d4d743\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.642174 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.642143 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpmzw\" (UniqueName: \"kubernetes.io/projected/db6abb4d-8ae3-4324-bc0c-f542c6d4d743-kube-api-access-tpmzw\") pod \"kuadrant-operator-controller-manager-86878c769f-z299b\" (UID: \"db6abb4d-8ae3-4324-bc0c-f542c6d4d743\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.758452 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.758356 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:34.910221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:34.910191 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b"] Jul 27 19:38:34.912057 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:38:34.912030 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb6abb4d_8ae3_4324_bc0c_f542c6d4d743.slice/crio-7fd8e319d99c9680049e2a6d533eb6606f9ca4237c4df05f9b3dd37160b5c5d0 WatchSource:0}: Error finding container 7fd8e319d99c9680049e2a6d533eb6606f9ca4237c4df05f9b3dd37160b5c5d0: Status 404 returned error can't find the container with id 7fd8e319d99c9680049e2a6d533eb6606f9ca4237c4df05f9b3dd37160b5c5d0 Jul 27 19:38:35.610935 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:35.610835 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" event={"ID":"db6abb4d-8ae3-4324-bc0c-f542c6d4d743","Type":"ContainerStarted","Data":"cd4e41f3a99c2261201f7b06193c433d74a51a834ed57972d2a1007da43deebe"} Jul 27 19:38:35.611442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:35.611111 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:38:35.611442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:35.611155 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" event={"ID":"db6abb4d-8ae3-4324-bc0c-f542c6d4d743","Type":"ContainerStarted","Data":"7fd8e319d99c9680049e2a6d533eb6606f9ca4237c4df05f9b3dd37160b5c5d0"} Jul 27 19:38:35.635372 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:35.635317 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" podStartSLOduration=1.635300642 podStartE2EDuration="1.635300642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:38:35.633726325 +0000 UTC m=+1288.628223302" watchObservedRunningTime="2026-07-27 19:38:35.635300642 +0000 UTC m=+1288.629797616" Jul 27 19:38:46.618818 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:38:46.618748 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z299b" Jul 27 19:42:07.605684 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:42:07.605655 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:42:07.608022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:42:07.607999 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:45:00.144195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.144149 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd"] Jul 27 19:45:00.148325 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.148301 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" Jul 27 19:45:00.151025 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.150996 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:45:00.159162 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.159137 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" Jul 27 19:45:00.163301 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.163271 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd"] Jul 27 19:45:00.319906 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.319879 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd"] Jul 27 19:45:00.321962 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:45:00.321933 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod400e38ae_bcd8_4f22_89a7_ca1eaf9694ed.slice/crio-2d5910e0129ae9dc0fa582b0243647352b2f9de98dcbd967ad5c435a48a5ad65 WatchSource:0}: Error finding container 2d5910e0129ae9dc0fa582b0243647352b2f9de98dcbd967ad5c435a48a5ad65: Status 404 returned error can't find the container with id 2d5910e0129ae9dc0fa582b0243647352b2f9de98dcbd967ad5c435a48a5ad65 Jul 27 19:45:00.803993 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.803897 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" event={"ID":"400e38ae-bcd8-4f22-89a7-ca1eaf9694ed","Type":"ContainerStarted","Data":"eed23322f93e9d3fe04bd5ae2f051a02ad2bc1d0bc4a0d0821fe9f2f21b0b4ee"} Jul 27 19:45:00.803993 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.803963 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" event={"ID":"400e38ae-bcd8-4f22-89a7-ca1eaf9694ed","Type":"ContainerStarted","Data":"2d5910e0129ae9dc0fa582b0243647352b2f9de98dcbd967ad5c435a48a5ad65"} Jul 27 19:45:00.821098 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:00.821042 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" podStartSLOduration=0.821021 podStartE2EDuration="821.021ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:45:00.819449476 +0000 UTC m=+1673.813946446" watchObservedRunningTime="2026-07-27 19:45:00.821021 +0000 UTC m=+1673.815517975" Jul 27 19:45:01.811365 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:01.811328 2566 generic.go:356] "Generic (PLEG): container finished" podID="400e38ae-bcd8-4f22-89a7-ca1eaf9694ed" containerID="eed23322f93e9d3fe04bd5ae2f051a02ad2bc1d0bc4a0d0821fe9f2f21b0b4ee" exitCode=0 Jul 27 19:45:01.811842 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:01.811413 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" event={"ID":"400e38ae-bcd8-4f22-89a7-ca1eaf9694ed","Type":"ContainerDied","Data":"eed23322f93e9d3fe04bd5ae2f051a02ad2bc1d0bc4a0d0821fe9f2f21b0b4ee"} Jul 27 19:45:02.971319 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:02.971289 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" Jul 27 19:45:03.824750 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:03.824666 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" Jul 27 19:45:03.824930 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:03.824661 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753025-d6tcd" event={"ID":"400e38ae-bcd8-4f22-89a7-ca1eaf9694ed","Type":"ContainerDied","Data":"2d5910e0129ae9dc0fa582b0243647352b2f9de98dcbd967ad5c435a48a5ad65"} Jul 27 19:45:03.824930 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:03.824819 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d5910e0129ae9dc0fa582b0243647352b2f9de98dcbd967ad5c435a48a5ad65" Jul 27 19:45:27.778856 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.778821 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q"] Jul 27 19:45:27.779309 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.779296 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="400e38ae-bcd8-4f22-89a7-ca1eaf9694ed" containerName="cleanup" Jul 27 19:45:27.779352 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.779309 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="400e38ae-bcd8-4f22-89a7-ca1eaf9694ed" containerName="cleanup" Jul 27 19:45:27.779387 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.779382 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="400e38ae-bcd8-4f22-89a7-ca1eaf9694ed" containerName="cleanup" Jul 27 19:45:27.782844 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.782826 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.785163 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.785138 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-68303af4-openshift-default-dockercfg-xtw85\"" Jul 27 19:45:27.793875 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.793845 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q"] Jul 27 19:45:27.910135 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910094 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-credential-socket\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910253 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-data\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910296 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-certs\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910331 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-token\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910505 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910398 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp6tw\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-kube-api-access-bp6tw\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910505 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910442 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-podinfo\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910505 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910478 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-socket\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910501 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istiod-ca-cert\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:27.910629 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:27.910577 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-envoy\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012042 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.011979 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-credential-socket\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012072 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-data\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012100 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-certs\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012129 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-token\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012163 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp6tw\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-kube-api-access-bp6tw\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012192 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-podinfo\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-socket\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012271 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012243 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istiod-ca-cert\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012305 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-envoy\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.012645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.012440 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-credential-socket\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.013091 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.013067 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-data\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.013497 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.013474 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-socket\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.013601 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.013567 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-certs\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.013775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.013729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istiod-ca-cert\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.015218 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.015197 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-envoy\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.015886 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.015863 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-podinfo\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.022733 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.022708 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp6tw\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-kube-api-access-bp6tw\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.023210 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.023186 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-token\") pod \"e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.095372 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.095265 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:28.251484 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.251427 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q"] Jul 27 19:45:28.254470 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:45:28.254446 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78b67140_f5c8_4aac_9066_1c1947e8bb7a.slice/crio-dd3041a02349d7afa664706d61b3079458ee00db03f65ee0f3cc36627c52bca7 WatchSource:0}: Error finding container dd3041a02349d7afa664706d61b3079458ee00db03f65ee0f3cc36627c52bca7: Status 404 returned error can't find the container with id dd3041a02349d7afa664706d61b3079458ee00db03f65ee0f3cc36627c52bca7 Jul 27 19:45:28.256907 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.256863 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:45:28.256982 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.256962 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:45:28.257027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.257008 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:45:28.982196 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.982149 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" event={"ID":"78b67140-f5c8-4aac-9066-1c1947e8bb7a","Type":"ContainerStarted","Data":"913cc8a151f1eb1b74549afdd193cbd30e703b8992c67024437ab0afe5a74d3c"} Jul 27 19:45:28.982654 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:28.982217 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" event={"ID":"78b67140-f5c8-4aac-9066-1c1947e8bb7a","Type":"ContainerStarted","Data":"dd3041a02349d7afa664706d61b3079458ee00db03f65ee0f3cc36627c52bca7"} Jul 27 19:45:29.020196 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:29.020145 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podStartSLOduration=2.020127662 podStartE2EDuration="2.020127662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:45:29.017732595 +0000 UTC m=+1702.012229571" watchObservedRunningTime="2026-07-27 19:45:29.020127662 +0000 UTC m=+1702.014624637" Jul 27 19:45:29.096358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:29.096317 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:29.101677 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:29.101648 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:29.987604 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:29.987575 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:29.988729 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:29.988710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:31.187756 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.187724 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7"] Jul 27 19:45:31.191577 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.191553 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.193965 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.193941 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-68303af4\"" Jul 27 19:45:31.194573 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.194555 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-68303af4-dockercfg-f4z4c\"" Jul 27 19:45:31.203407 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.203372 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7"] Jul 27 19:45:31.214855 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.214819 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4"] Jul 27 19:45:31.218780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.218729 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.229992 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.229958 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4"] Jul 27 19:45:31.248176 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.248132 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67n5\" (UniqueName: \"kubernetes.io/projected/82d1d488-89ad-4f16-ac3a-7527680368dc-kube-api-access-l67n5\") pod \"payload-processing-e2e-ait-68303af4-646f9df866-77pp4\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.248176 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.248172 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1d488-89ad-4f16-ac3a-7527680368dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-68303af4-646f9df866-77pp4\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.248455 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.248297 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.248455 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.248321 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd4nr\" (UniqueName: \"kubernetes.io/projected/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-kube-api-access-wd4nr\") pod \"payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.349482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.349434 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.349482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.349471 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd4nr\" (UniqueName: \"kubernetes.io/projected/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-kube-api-access-wd4nr\") pod \"payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.349754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.349530 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67n5\" (UniqueName: \"kubernetes.io/projected/82d1d488-89ad-4f16-ac3a-7527680368dc-kube-api-access-l67n5\") pod \"payload-processing-e2e-ait-68303af4-646f9df866-77pp4\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.349754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.349553 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1d488-89ad-4f16-ac3a-7527680368dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-68303af4-646f9df866-77pp4\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.350207 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.350182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.350339 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.350318 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1d488-89ad-4f16-ac3a-7527680368dc-plugins-config-volume\") pod \"payload-processing-e2e-ait-68303af4-646f9df866-77pp4\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.358916 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.358876 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd4nr\" (UniqueName: \"kubernetes.io/projected/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-kube-api-access-wd4nr\") pod \"payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.359081 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.358954 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67n5\" (UniqueName: \"kubernetes.io/projected/82d1d488-89ad-4f16-ac3a-7527680368dc-kube-api-access-l67n5\") pod \"payload-processing-e2e-ait-68303af4-646f9df866-77pp4\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.504724 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.504625 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:31.532119 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.532082 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:31.685235 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.685201 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7"] Jul 27 19:45:31.688227 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:45:31.688195 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b5b3970_c6c9_4a1e_a446_e2bca3796cdb.slice/crio-8cc460540f53849fd343746d815758286d5f7578841930023a68960fbb43a953 WatchSource:0}: Error finding container 8cc460540f53849fd343746d815758286d5f7578841930023a68960fbb43a953: Status 404 returned error can't find the container with id 8cc460540f53849fd343746d815758286d5f7578841930023a68960fbb43a953 Jul 27 19:45:31.918225 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:31.918200 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4"] Jul 27 19:45:31.921319 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:45:31.921289 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82d1d488_89ad_4f16_ac3a_7527680368dc.slice/crio-c21c3521f9f42984ba729d8194eb5ccb8bbb13c5c7661a5d1d5d2466457f6ad0 WatchSource:0}: Error finding container c21c3521f9f42984ba729d8194eb5ccb8bbb13c5c7661a5d1d5d2466457f6ad0: Status 404 returned error can't find the container with id c21c3521f9f42984ba729d8194eb5ccb8bbb13c5c7661a5d1d5d2466457f6ad0 Jul 27 19:45:32.000959 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.000921 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" event={"ID":"82d1d488-89ad-4f16-ac3a-7527680368dc","Type":"ContainerStarted","Data":"dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea"} Jul 27 19:45:32.001170 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.000994 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:32.001170 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.001018 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" event={"ID":"82d1d488-89ad-4f16-ac3a-7527680368dc","Type":"ContainerStarted","Data":"c21c3521f9f42984ba729d8194eb5ccb8bbb13c5c7661a5d1d5d2466457f6ad0"} Jul 27 19:45:32.002832 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.002795 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" event={"ID":"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb","Type":"ContainerStarted","Data":"1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f"} Jul 27 19:45:32.002989 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.002853 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" event={"ID":"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb","Type":"ContainerStarted","Data":"8cc460540f53849fd343746d815758286d5f7578841930023a68960fbb43a953"} Jul 27 19:45:32.002989 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.002916 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:32.029912 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.029844 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" podStartSLOduration=1.029826476 podStartE2EDuration="1.029826476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:45:32.025392748 +0000 UTC m=+1705.019889724" watchObservedRunningTime="2026-07-27 19:45:32.029826476 +0000 UTC m=+1705.024323451" Jul 27 19:45:32.051586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:32.051525 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" podStartSLOduration=1.051506704 podStartE2EDuration="1.051506704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:45:32.048446206 +0000 UTC m=+1705.042943193" watchObservedRunningTime="2026-07-27 19:45:32.051506704 +0000 UTC m=+1705.046003678" Jul 27 19:45:43.010587 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:43.010525 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" podUID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.63:9004: connect: connection refused" Jul 27 19:45:43.014026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:43.013991 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:45:46.028971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:46.028928 2566 patch_prober.go:28] interesting pod/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:45:46.029432 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:46.029001 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:45:47.092348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:47.092258 2566 generic.go:356] "Generic (PLEG): container finished" podID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerID="913cc8a151f1eb1b74549afdd193cbd30e703b8992c67024437ab0afe5a74d3c" exitCode=0 Jul 27 19:45:47.092348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:47.092319 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" event={"ID":"78b67140-f5c8-4aac-9066-1c1947e8bb7a","Type":"ContainerDied","Data":"913cc8a151f1eb1b74549afdd193cbd30e703b8992c67024437ab0afe5a74d3c"} Jul 27 19:45:47.092804 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:47.092750 2566 scope.go:117] "RemoveContainer" containerID="913cc8a151f1eb1b74549afdd193cbd30e703b8992c67024437ab0afe5a74d3c" Jul 27 19:45:47.093780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:47.093731 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:45:47.093868 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:47.093853 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:45:47.093906 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:47.093896 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:45:48.099524 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:48.099481 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" event={"ID":"78b67140-f5c8-4aac-9066-1c1947e8bb7a","Type":"ContainerStarted","Data":"2ec873b6d2d97dc107598569ab562a1678ae3392ca239ad33419406509e35fd6"} Jul 27 19:45:49.095621 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:49.095575 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:49.096061 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:49.095935 2566 patch_prober.go:28] interesting pod/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:45:49.096061 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:49.095983 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:45:50.096816 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:50.096736 2566 patch_prober.go:28] interesting pod/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:45:50.097237 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:50.096835 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:45:51.096716 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:51.096681 2566 patch_prober.go:28] interesting pod/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:45:51.096935 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:51.096739 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:45:52.095939 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:52.095893 2566 patch_prober.go:28] interesting pod/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 27 19:45:52.096151 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:52.095961 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 27 19:45:53.010069 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:53.010036 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:45:53.099699 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:53.099666 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:53.100006 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:53.099982 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:45:53.100817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:45:53.100798 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:46:06.505994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.505955 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4"] Jul 27 19:46:06.506550 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.506182 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" podUID="82d1d488-89ad-4f16-ac3a-7527680368dc" containerName="payload-processing" containerID="cri-o://dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea" gracePeriod=30 Jul 27 19:46:06.513570 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.513522 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7"] Jul 27 19:46:06.514090 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.514040 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" podUID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" containerName="payload-pre-processing" containerID="cri-o://1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f" gracePeriod=30 Jul 27 19:46:06.813564 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.813531 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:46:06.818033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.818004 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:46:06.933739 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.933691 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1d488-89ad-4f16-ac3a-7527680368dc-plugins-config-volume\") pod \"82d1d488-89ad-4f16-ac3a-7527680368dc\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " Jul 27 19:46:06.933968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.933752 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l67n5\" (UniqueName: \"kubernetes.io/projected/82d1d488-89ad-4f16-ac3a-7527680368dc-kube-api-access-l67n5\") pod \"82d1d488-89ad-4f16-ac3a-7527680368dc\" (UID: \"82d1d488-89ad-4f16-ac3a-7527680368dc\") " Jul 27 19:46:06.933968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.933811 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-plugins-config-volume\") pod \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " Jul 27 19:46:06.933968 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.933852 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wd4nr\" (UniqueName: \"kubernetes.io/projected/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-kube-api-access-wd4nr\") pod \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\" (UID: \"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb\") " Jul 27 19:46:06.934165 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.934140 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82d1d488-89ad-4f16-ac3a-7527680368dc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82d1d488-89ad-4f16-ac3a-7527680368dc" (UID: "82d1d488-89ad-4f16-ac3a-7527680368dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:46:06.934226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.934202 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" (UID: "0b5b3970-c6c9-4a1e-a446-e2bca3796cdb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:46:06.936092 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.936059 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-kube-api-access-wd4nr" (OuterVolumeSpecName: "kube-api-access-wd4nr") pod "0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" (UID: "0b5b3970-c6c9-4a1e-a446-e2bca3796cdb"). InnerVolumeSpecName "kube-api-access-wd4nr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:06.936197 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:06.936091 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82d1d488-89ad-4f16-ac3a-7527680368dc-kube-api-access-l67n5" (OuterVolumeSpecName: "kube-api-access-l67n5") pod "82d1d488-89ad-4f16-ac3a-7527680368dc" (UID: "82d1d488-89ad-4f16-ac3a-7527680368dc"). InnerVolumeSpecName "kube-api-access-l67n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:07.035553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.035454 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82d1d488-89ad-4f16-ac3a-7527680368dc-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:07.035553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.035485 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67n5\" (UniqueName: \"kubernetes.io/projected/82d1d488-89ad-4f16-ac3a-7527680368dc-kube-api-access-l67n5\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:07.035553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.035494 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:07.035553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.035507 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd4nr\" (UniqueName: \"kubernetes.io/projected/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb-kube-api-access-wd4nr\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:07.219389 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.219349 2566 generic.go:356] "Generic (PLEG): container finished" podID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" containerID="1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f" exitCode=0 Jul 27 19:46:07.219589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.219419 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" Jul 27 19:46:07.219589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.219440 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" event={"ID":"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb","Type":"ContainerDied","Data":"1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f"} Jul 27 19:46:07.219589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.219537 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7" event={"ID":"0b5b3970-c6c9-4a1e-a446-e2bca3796cdb","Type":"ContainerDied","Data":"8cc460540f53849fd343746d815758286d5f7578841930023a68960fbb43a953"} Jul 27 19:46:07.219589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.219567 2566 scope.go:117] "RemoveContainer" containerID="1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f" Jul 27 19:46:07.221165 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.221137 2566 generic.go:356] "Generic (PLEG): container finished" podID="82d1d488-89ad-4f16-ac3a-7527680368dc" containerID="dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea" exitCode=0 Jul 27 19:46:07.221292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.221256 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" event={"ID":"82d1d488-89ad-4f16-ac3a-7527680368dc","Type":"ContainerDied","Data":"dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea"} Jul 27 19:46:07.221357 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.221302 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" event={"ID":"82d1d488-89ad-4f16-ac3a-7527680368dc","Type":"ContainerDied","Data":"c21c3521f9f42984ba729d8194eb5ccb8bbb13c5c7661a5d1d5d2466457f6ad0"} Jul 27 19:46:07.221357 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.221341 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4" Jul 27 19:46:07.229525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.229504 2566 scope.go:117] "RemoveContainer" containerID="1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f" Jul 27 19:46:07.229875 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:46:07.229849 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f\": container with ID starting with 1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f not found: ID does not exist" containerID="1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f" Jul 27 19:46:07.229975 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.229887 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f"} err="failed to get container status \"1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f\": rpc error: code = NotFound desc = could not find container \"1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f\": container with ID starting with 1f42e4d2d338b11a256baad5884541bbd32aec755eab8dd4e9f169417a76e72f not found: ID does not exist" Jul 27 19:46:07.229975 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.229910 2566 scope.go:117] "RemoveContainer" containerID="dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea" Jul 27 19:46:07.238858 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.238836 2566 scope.go:117] "RemoveContainer" containerID="dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea" Jul 27 19:46:07.239149 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:46:07.239129 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea\": container with ID starting with dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea not found: ID does not exist" containerID="dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea" Jul 27 19:46:07.239224 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.239158 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea"} err="failed to get container status \"dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea\": rpc error: code = NotFound desc = could not find container \"dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea\": container with ID starting with dfbcde6b2e2a5a56eb777663bb1f0ac0b53f8d7b7a2bd2cc33342aa82d822cea not found: ID does not exist" Jul 27 19:46:07.249695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.249666 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7"] Jul 27 19:46:07.255167 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.255141 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-68303af4-577d645b8d-4prd7"] Jul 27 19:46:07.267213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.267179 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4"] Jul 27 19:46:07.271855 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.271825 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-68303af4-646f9df866-77pp4"] Jul 27 19:46:07.547151 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.547115 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" path="/var/lib/kubelet/pods/0b5b3970-c6c9-4a1e-a446-e2bca3796cdb/volumes" Jul 27 19:46:07.547669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:07.547647 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82d1d488-89ad-4f16-ac3a-7527680368dc" path="/var/lib/kubelet/pods/82d1d488-89ad-4f16-ac3a-7527680368dc/volumes" Jul 27 19:46:11.148484 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:11.148437 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q"] Jul 27 19:46:11.148993 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:11.148720 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" containerID="cri-o://2ec873b6d2d97dc107598569ab562a1678ae3392ca239ad33419406509e35fd6" gracePeriod=30 Jul 27 19:46:12.145659 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.145615 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb"] Jul 27 19:46:12.146270 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.146247 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82d1d488-89ad-4f16-ac3a-7527680368dc" containerName="payload-processing" Jul 27 19:46:12.146270 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.146271 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="82d1d488-89ad-4f16-ac3a-7527680368dc" containerName="payload-processing" Jul 27 19:46:12.146472 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.146291 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" containerName="payload-pre-processing" Jul 27 19:46:12.146472 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.146297 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" containerName="payload-pre-processing" Jul 27 19:46:12.146472 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.146425 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b5b3970-c6c9-4a1e-a446-e2bca3796cdb" containerName="payload-pre-processing" Jul 27 19:46:12.146472 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.146440 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="82d1d488-89ad-4f16-ac3a-7527680368dc" containerName="payload-processing" Jul 27 19:46:12.149470 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.149449 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.153156 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.153138 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-a312c841-gw-openshift-default-dockercfg-clz29\"" Jul 27 19:46:12.164628 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.164598 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb"] Jul 27 19:46:12.181672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181642 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-credential-socket\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.181864 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181682 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-socket\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.181864 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181713 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-token\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.181864 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181734 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-data\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.181864 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181831 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-envoy\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.182109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181892 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-podinfo\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.182109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181952 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwhcz\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-kube-api-access-vwhcz\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.182109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.181998 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-certs\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.182109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.182021 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49c790-21af-4755-9d76-fc0c29d3da4c-istiod-ca-cert\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283202 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-podinfo\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283223 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwhcz\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-kube-api-access-vwhcz\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283253 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-certs\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283279 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49c790-21af-4755-9d76-fc0c29d3da4c-istiod-ca-cert\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283324 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-credential-socket\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283640 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283472 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-socket\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283640 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283619 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-token\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283753 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-data\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283753 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283718 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-envoy\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283753 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283734 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-certs\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.283958 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283808 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-credential-socket\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.284016 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.283974 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-socket\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.284157 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.284135 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49c790-21af-4755-9d76-fc0c29d3da4c-istiod-ca-cert\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.284157 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.284146 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-data\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.285894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.285865 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-envoy\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.286054 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.286036 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-podinfo\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.291964 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.291939 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-token\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.292088 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.292061 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwhcz\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-kube-api-access-vwhcz\") pod \"e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.461176 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.461138 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:12.624887 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.624859 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb"] Jul 27 19:46:12.627069 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:46:12.627039 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf49c790_21af_4755_9d76_fc0c29d3da4c.slice/crio-a689cb933dda2f8a3cce1e95283576275031bcac4e476fa8a75a16caa36992dd WatchSource:0}: Error finding container a689cb933dda2f8a3cce1e95283576275031bcac4e476fa8a75a16caa36992dd: Status 404 returned error can't find the container with id a689cb933dda2f8a3cce1e95283576275031bcac4e476fa8a75a16caa36992dd Jul 27 19:46:12.629359 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.629324 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:12.629446 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.629406 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:12.629446 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:12.629438 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:13.261748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:13.261707 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" event={"ID":"af49c790-21af-4755-9d76-fc0c29d3da4c","Type":"ContainerStarted","Data":"1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9"} Jul 27 19:46:13.262186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:13.261799 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" event={"ID":"af49c790-21af-4755-9d76-fc0c29d3da4c","Type":"ContainerStarted","Data":"a689cb933dda2f8a3cce1e95283576275031bcac4e476fa8a75a16caa36992dd"} Jul 27 19:46:13.284849 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:13.284789 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podStartSLOduration=1.284771864 podStartE2EDuration="1.284771864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:13.283175621 +0000 UTC m=+1746.277672597" watchObservedRunningTime="2026-07-27 19:46:13.284771864 +0000 UTC m=+1746.279268830" Jul 27 19:46:13.461641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:13.461603 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:13.467223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:13.467196 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:14.267626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:14.267596 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:14.268775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:14.268743 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:15.566565 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.566523 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d"] Jul 27 19:46:15.570668 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.570643 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.573109 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.573080 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a312c841-dockercfg-8f89g\"" Jul 27 19:46:15.573260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.573088 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a312c841\"" Jul 27 19:46:15.584594 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.584567 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d"] Jul 27 19:46:15.617737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.617701 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f79067d-3630-438f-b3a8-cb6a1279fcc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.617977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.617953 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w95s8\" (UniqueName: \"kubernetes.io/projected/2f79067d-3630-438f-b3a8-cb6a1279fcc4-kube-api-access-w95s8\") pod \"payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.719689 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.719638 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f79067d-3630-438f-b3a8-cb6a1279fcc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.719904 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.719779 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w95s8\" (UniqueName: \"kubernetes.io/projected/2f79067d-3630-438f-b3a8-cb6a1279fcc4-kube-api-access-w95s8\") pod \"payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.720573 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.720547 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f79067d-3630-438f-b3a8-cb6a1279fcc4-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.728002 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.727970 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w95s8\" (UniqueName: \"kubernetes.io/projected/2f79067d-3630-438f-b3a8-cb6a1279fcc4-kube-api-access-w95s8\") pod \"payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:15.882999 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:15.882895 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:16.046998 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.046969 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d"] Jul 27 19:46:16.049722 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:46:16.049691 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f79067d_3630_438f_b3a8_cb6a1279fcc4.slice/crio-dbd87b4389df3801150bd83b9ef5f249c95f86b86f0d5f750e549fc09713b81d WatchSource:0}: Error finding container dbd87b4389df3801150bd83b9ef5f249c95f86b86f0d5f750e549fc09713b81d: Status 404 returned error can't find the container with id dbd87b4389df3801150bd83b9ef5f249c95f86b86f0d5f750e549fc09713b81d Jul 27 19:46:16.280815 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.280777 2566 generic.go:356] "Generic (PLEG): container finished" podID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerID="2ec873b6d2d97dc107598569ab562a1678ae3392ca239ad33419406509e35fd6" exitCode=0 Jul 27 19:46:16.280815 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.280794 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" event={"ID":"78b67140-f5c8-4aac-9066-1c1947e8bb7a","Type":"ContainerDied","Data":"2ec873b6d2d97dc107598569ab562a1678ae3392ca239ad33419406509e35fd6"} Jul 27 19:46:16.281029 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.280857 2566 scope.go:117] "RemoveContainer" containerID="913cc8a151f1eb1b74549afdd193cbd30e703b8992c67024437ab0afe5a74d3c" Jul 27 19:46:16.282600 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.282559 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" event={"ID":"2f79067d-3630-438f-b3a8-cb6a1279fcc4","Type":"ContainerStarted","Data":"a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86"} Jul 27 19:46:16.282600 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.282603 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" event={"ID":"2f79067d-3630-438f-b3a8-cb6a1279fcc4","Type":"ContainerStarted","Data":"dbd87b4389df3801150bd83b9ef5f249c95f86b86f0d5f750e549fc09713b81d"} Jul 27 19:46:16.282869 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.282660 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:16.302016 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.301966 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" podStartSLOduration=1.301948873 podStartE2EDuration="1.301948873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:16.299998976 +0000 UTC m=+1749.294495950" watchObservedRunningTime="2026-07-27 19:46:16.301948873 +0000 UTC m=+1749.296445847" Jul 27 19:46:16.398898 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.398871 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:46:16.427183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427145 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-podinfo\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427383 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427206 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-socket\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427383 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427264 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-token\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427383 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427292 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-credential-socket\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427383 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427344 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-certs\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427638 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427397 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istiod-ca-cert\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427638 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427432 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-data\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427638 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427497 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-envoy\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427638 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427564 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp6tw\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-kube-api-access-bp6tw\") pod \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\" (UID: \"78b67140-f5c8-4aac-9066-1c1947e8bb7a\") " Jul 27 19:46:16.427853 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427706 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:16.428502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.427997 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.428502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.428307 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:46:16.428502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.428403 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:16.428502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.428460 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:16.428929 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.428876 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-data" (OuterVolumeSpecName: "istio-data") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:16.430555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.430509 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-kube-api-access-bp6tw" (OuterVolumeSpecName: "kube-api-access-bp6tw") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "kube-api-access-bp6tw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:16.430694 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.430598 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:46:16.430922 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.430896 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:46:16.431142 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.431116 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-token" (OuterVolumeSpecName: "istio-token") pod "78b67140-f5c8-4aac-9066-1c1947e8bb7a" (UID: "78b67140-f5c8-4aac-9066-1c1947e8bb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:16.529153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529110 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529148 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529159 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529169 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp6tw\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-kube-api-access-bp6tw\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529184 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529194 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78b67140-f5c8-4aac-9066-1c1947e8bb7a-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529205 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:16.529410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:16.529216 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78b67140-f5c8-4aac-9066-1c1947e8bb7a-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:17.289679 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:17.289644 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" Jul 27 19:46:17.290160 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:17.289684 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q" event={"ID":"78b67140-f5c8-4aac-9066-1c1947e8bb7a","Type":"ContainerDied","Data":"dd3041a02349d7afa664706d61b3079458ee00db03f65ee0f3cc36627c52bca7"} Jul 27 19:46:17.290160 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:17.289740 2566 scope.go:117] "RemoveContainer" containerID="2ec873b6d2d97dc107598569ab562a1678ae3392ca239ad33419406509e35fd6" Jul 27 19:46:17.316608 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:17.316569 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q"] Jul 27 19:46:17.320526 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:17.320489 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-68303af4-openshift-default-5cbf67dfb6-l7m4q"] Jul 27 19:46:17.546604 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:17.546518 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" path="/var/lib/kubelet/pods/78b67140-f5c8-4aac-9066-1c1947e8bb7a/volumes" Jul 27 19:46:27.292949 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:27.292918 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:29.711689 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:29.710783 2566 patch_prober.go:28] interesting pod/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": read tcp 10.132.0.2:36574->10.132.0.65:15021: read: connection reset by peer" start-of-body= Jul 27 19:46:29.711689 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:29.710923 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": read tcp 10.132.0.2:36574->10.132.0.65:15021: read: connection reset by peer" Jul 27 19:46:30.386689 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:30.386656 2566 generic.go:356] "Generic (PLEG): container finished" podID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerID="1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9" exitCode=0 Jul 27 19:46:30.386899 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:30.386727 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" event={"ID":"af49c790-21af-4755-9d76-fc0c29d3da4c","Type":"ContainerDied","Data":"1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9"} Jul 27 19:46:30.387199 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:30.387181 2566 scope.go:117] "RemoveContainer" containerID="1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9" Jul 27 19:46:30.388311 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:30.388279 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:30.388405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:30.388348 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:30.388405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:30.388379 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:31.394639 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:31.394605 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" event={"ID":"af49c790-21af-4755-9d76-fc0c29d3da4c","Type":"ContainerStarted","Data":"7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293"} Jul 27 19:46:31.461571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:31.461538 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:31.461946 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:31.461916 2566 patch_prober.go:28] interesting pod/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:46:31.462012 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:31.461990 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:46:32.461684 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:32.461644 2566 patch_prober.go:28] interesting pod/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:46:32.462119 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:32.461702 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:46:33.462068 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:33.462030 2566 patch_prober.go:28] interesting pod/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:46:33.462477 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:33.462090 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:46:34.462035 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:34.462001 2566 patch_prober.go:28] interesting pod/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" start-of-body= Jul 27 19:46:34.462246 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:34.462060 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.65:15021/healthz/ready\": dial tcp 10.132.0.65:15021: connect: connection refused" Jul 27 19:46:35.466067 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:35.466031 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:35.466565 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:35.466385 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:35.467142 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:35.467121 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:46:39.714961 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.714922 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn"] Jul 27 19:46:39.715446 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.715431 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" Jul 27 19:46:39.715497 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.715449 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" Jul 27 19:46:39.715574 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.715563 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" Jul 27 19:46:39.715611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.715580 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" Jul 27 19:46:39.718854 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.718836 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:39.721494 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.721463 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:46:39.727367 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.727286 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn"] Jul 27 19:46:39.881828 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.881779 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/242c1060-004a-487d-9f7d-fd324ab73965-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn\" (UID: \"242c1060-004a-487d-9f7d-fd324ab73965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:39.983057 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.982955 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/242c1060-004a-487d-9f7d-fd324ab73965-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn\" (UID: \"242c1060-004a-487d-9f7d-fd324ab73965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:39.983664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:39.983637 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/242c1060-004a-487d-9f7d-fd324ab73965-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn\" (UID: \"242c1060-004a-487d-9f7d-fd324ab73965\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:40.033057 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:40.033022 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:40.191041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:40.191004 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn"] Jul 27 19:46:40.193197 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:46:40.193165 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod242c1060_004a_487d_9f7d_fd324ab73965.slice/crio-785aceab1f8df26776f4514529878d5255934f92fa13a3ba28b783480a3eef33 WatchSource:0}: Error finding container 785aceab1f8df26776f4514529878d5255934f92fa13a3ba28b783480a3eef33: Status 404 returned error can't find the container with id 785aceab1f8df26776f4514529878d5255934f92fa13a3ba28b783480a3eef33 Jul 27 19:46:40.452094 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:40.452054 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" event={"ID":"242c1060-004a-487d-9f7d-fd324ab73965","Type":"ContainerStarted","Data":"785aceab1f8df26776f4514529878d5255934f92fa13a3ba28b783480a3eef33"} Jul 27 19:46:41.459777 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:41.459726 2566 generic.go:356] "Generic (PLEG): container finished" podID="242c1060-004a-487d-9f7d-fd324ab73965" containerID="56c1738bfd3e4c7909c00f9f3807092d373878e31290cf3576b1b8ba7ba355d8" exitCode=0 Jul 27 19:46:41.460193 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:41.459815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" event={"ID":"242c1060-004a-487d-9f7d-fd324ab73965","Type":"ContainerDied","Data":"56c1738bfd3e4c7909c00f9f3807092d373878e31290cf3576b1b8ba7ba355d8"} Jul 27 19:46:42.641248 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:42.641219 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:42.706534 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:42.706495 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/242c1060-004a-487d-9f7d-fd324ab73965-maas-api-service-ca\") pod \"242c1060-004a-487d-9f7d-fd324ab73965\" (UID: \"242c1060-004a-487d-9f7d-fd324ab73965\") " Jul 27 19:46:42.706967 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:42.706947 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/242c1060-004a-487d-9f7d-fd324ab73965-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "242c1060-004a-487d-9f7d-fd324ab73965" (UID: "242c1060-004a-487d-9f7d-fd324ab73965"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:46:42.808082 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:42.807976 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/242c1060-004a-487d-9f7d-fd324ab73965-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:43.473302 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:43.473269 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" event={"ID":"242c1060-004a-487d-9f7d-fd324ab73965","Type":"ContainerDied","Data":"785aceab1f8df26776f4514529878d5255934f92fa13a3ba28b783480a3eef33"} Jul 27 19:46:43.473460 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:43.473316 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785aceab1f8df26776f4514529878d5255934f92fa13a3ba28b783480a3eef33" Jul 27 19:46:43.473460 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:43.473279 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn" Jul 27 19:46:50.207064 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.207026 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d"] Jul 27 19:46:50.208098 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.207302 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" podUID="2f79067d-3630-438f-b3a8-cb6a1279fcc4" containerName="payload-pre-processing" containerID="cri-o://a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86" gracePeriod=30 Jul 27 19:46:50.488193 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.488164 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:50.516318 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.516281 2566 generic.go:356] "Generic (PLEG): container finished" podID="2f79067d-3630-438f-b3a8-cb6a1279fcc4" containerID="a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86" exitCode=0 Jul 27 19:46:50.516318 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.516325 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" Jul 27 19:46:50.516537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.516361 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" event={"ID":"2f79067d-3630-438f-b3a8-cb6a1279fcc4","Type":"ContainerDied","Data":"a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86"} Jul 27 19:46:50.516537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.516412 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d" event={"ID":"2f79067d-3630-438f-b3a8-cb6a1279fcc4","Type":"ContainerDied","Data":"dbd87b4389df3801150bd83b9ef5f249c95f86b86f0d5f750e549fc09713b81d"} Jul 27 19:46:50.516537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.516441 2566 scope.go:117] "RemoveContainer" containerID="a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86" Jul 27 19:46:50.528895 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.528835 2566 scope.go:117] "RemoveContainer" containerID="a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86" Jul 27 19:46:50.529241 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:46:50.529213 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86\": container with ID starting with a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86 not found: ID does not exist" containerID="a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86" Jul 27 19:46:50.529347 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.529245 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86"} err="failed to get container status \"a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86\": rpc error: code = NotFound desc = could not find container \"a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86\": container with ID starting with a80c8174040d37f2af729ab6148d1cb8321c4cf97b117244e368fddcb7168a86 not found: ID does not exist" Jul 27 19:46:50.587574 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.587527 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w95s8\" (UniqueName: \"kubernetes.io/projected/2f79067d-3630-438f-b3a8-cb6a1279fcc4-kube-api-access-w95s8\") pod \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " Jul 27 19:46:50.587750 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.587609 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f79067d-3630-438f-b3a8-cb6a1279fcc4-plugins-config-volume\") pod \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\" (UID: \"2f79067d-3630-438f-b3a8-cb6a1279fcc4\") " Jul 27 19:46:50.588363 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.588023 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f79067d-3630-438f-b3a8-cb6a1279fcc4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f79067d-3630-438f-b3a8-cb6a1279fcc4" (UID: "2f79067d-3630-438f-b3a8-cb6a1279fcc4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:46:50.588363 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.588138 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f79067d-3630-438f-b3a8-cb6a1279fcc4-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:50.589865 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.589844 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f79067d-3630-438f-b3a8-cb6a1279fcc4-kube-api-access-w95s8" (OuterVolumeSpecName: "kube-api-access-w95s8") pod "2f79067d-3630-438f-b3a8-cb6a1279fcc4" (UID: "2f79067d-3630-438f-b3a8-cb6a1279fcc4"). InnerVolumeSpecName "kube-api-access-w95s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:46:50.689442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.689408 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w95s8\" (UniqueName: \"kubernetes.io/projected/2f79067d-3630-438f-b3a8-cb6a1279fcc4-kube-api-access-w95s8\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:46:50.842371 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.842334 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d"] Jul 27 19:46:50.847329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:50.847289 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a312c841-7fbb5dd4fb-xvh5d"] Jul 27 19:46:51.547103 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:51.547071 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f79067d-3630-438f-b3a8-cb6a1279fcc4" path="/var/lib/kubelet/pods/2f79067d-3630-438f-b3a8-cb6a1279fcc4/volumes" Jul 27 19:46:54.804020 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:54.803973 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn"] Jul 27 19:46:54.808348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:54.808315 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-a312c841-q5dkn"] Jul 27 19:46:54.937543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:54.937500 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb"] Jul 27 19:46:54.937740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:54.937718 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" containerID="cri-o://7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293" gracePeriod=30 Jul 27 19:46:55.547867 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.547830 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="242c1060-004a-487d-9f7d-fd324ab73965" path="/var/lib/kubelet/pods/242c1060-004a-487d-9f7d-fd324ab73965/volumes" Jul 27 19:46:55.670156 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670127 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2"] Jul 27 19:46:55.670625 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670611 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="242c1060-004a-487d-9f7d-fd324ab73965" containerName="revoke-keys" Jul 27 19:46:55.670695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670627 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="242c1060-004a-487d-9f7d-fd324ab73965" containerName="revoke-keys" Jul 27 19:46:55.670695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670639 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" Jul 27 19:46:55.670695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670645 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78b67140-f5c8-4aac-9066-1c1947e8bb7a" containerName="istio-proxy" Jul 27 19:46:55.670695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670665 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f79067d-3630-438f-b3a8-cb6a1279fcc4" containerName="payload-pre-processing" Jul 27 19:46:55.670695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670671 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f79067d-3630-438f-b3a8-cb6a1279fcc4" containerName="payload-pre-processing" Jul 27 19:46:55.670884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670739 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="242c1060-004a-487d-9f7d-fd324ab73965" containerName="revoke-keys" Jul 27 19:46:55.670884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.670749 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f79067d-3630-438f-b3a8-cb6a1279fcc4" containerName="payload-pre-processing" Jul 27 19:46:55.675325 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.675306 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.678261 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.678237 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0b0189d4-openshift-default-dockercfg-wmvrd\"" Jul 27 19:46:55.689390 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.689353 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2"] Jul 27 19:46:55.742246 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742207 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e8c485-44c7-4694-bd85-a06a26578f37-istiod-ca-cert\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742260 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-envoy\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742327 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-credential-socket\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742382 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e8c485-44c7-4694-bd85-a06a26578f37-istio-podinfo\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742567 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742429 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-istio-token\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742567 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742457 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-certs\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742567 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742484 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsz7\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-kube-api-access-tjsz7\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742724 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-data\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.742724 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.742670 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-socket\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.843539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843432 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-credential-socket\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.843539 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843499 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e8c485-44c7-4694-bd85-a06a26578f37-istio-podinfo\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843550 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-istio-token\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843577 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-certs\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843605 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsz7\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-kube-api-access-tjsz7\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843822 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-data\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843827 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-credential-socket\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843914 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-socket\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.843968 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-certs\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.844000 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e8c485-44c7-4694-bd85-a06a26578f37-istiod-ca-cert\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844073 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.844041 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-envoy\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844468 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.844152 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-socket\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844468 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.844219 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-data\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.844672 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.844649 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e8c485-44c7-4694-bd85-a06a26578f37-istiod-ca-cert\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.846087 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.846064 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-envoy\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.846217 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.846200 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e8c485-44c7-4694-bd85-a06a26578f37-istio-podinfo\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.853136 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.853100 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-istio-token\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.853280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.853255 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsz7\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-kube-api-access-tjsz7\") pod \"e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:55.986401 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:55.986362 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:56.147058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.147023 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2"] Jul 27 19:46:56.149262 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:46:56.149229 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21e8c485_44c7_4694_bd85_a06a26578f37.slice/crio-e790a9ea1295d6efeeb3515292c80047520946c1c31dad06bcfedd2e7f5eb5d2 WatchSource:0}: Error finding container e790a9ea1295d6efeeb3515292c80047520946c1c31dad06bcfedd2e7f5eb5d2: Status 404 returned error can't find the container with id e790a9ea1295d6efeeb3515292c80047520946c1c31dad06bcfedd2e7f5eb5d2 Jul 27 19:46:56.151383 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.151347 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:56.151477 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.151416 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:56.151477 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.151444 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:46:56.554031 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.553983 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" event={"ID":"21e8c485-44c7-4694-bd85-a06a26578f37","Type":"ContainerStarted","Data":"4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1"} Jul 27 19:46:56.554245 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.554048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" event={"ID":"21e8c485-44c7-4694-bd85-a06a26578f37","Type":"ContainerStarted","Data":"e790a9ea1295d6efeeb3515292c80047520946c1c31dad06bcfedd2e7f5eb5d2"} Jul 27 19:46:56.577691 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.577631 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podStartSLOduration=1.57761445 podStartE2EDuration="1.57761445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:46:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:46:56.576856508 +0000 UTC m=+1789.571353508" watchObservedRunningTime="2026-07-27 19:46:56.57761445 +0000 UTC m=+1789.572111487" Jul 27 19:46:56.986862 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:56.986820 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:57.992382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:57.992347 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:58.566005 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:58.565966 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:46:58.566912 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:46:58.566879 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:00.217913 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.217886 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:47:00.293090 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293049 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-socket\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293090 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293091 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-podinfo\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293333 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293128 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49c790-21af-4755-9d76-fc0c29d3da4c-istiod-ca-cert\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293333 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293150 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-token\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293333 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293174 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vwhcz\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-kube-api-access-vwhcz\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293333 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293198 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-envoy\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293333 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293284 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-certs\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293338 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-credential-socket\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293392 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-data\") pod \"af49c790-21af-4755-9d76-fc0c29d3da4c\" (UID: \"af49c790-21af-4755-9d76-fc0c29d3da4c\") " Jul 27 19:47:00.293599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293420 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:00.293599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293530 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af49c790-21af-4755-9d76-fc0c29d3da4c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:47:00.293847 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293669 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:00.293847 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.293837 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:00.294114 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.294087 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.294114 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.294108 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.294254 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.294122 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.294254 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.294135 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af49c790-21af-4755-9d76-fc0c29d3da4c-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.294364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.294308 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-data" (OuterVolumeSpecName: "istio-data") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:00.295613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.295588 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-token" (OuterVolumeSpecName: "istio-token") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:47:00.296045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.296013 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:47:00.296132 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.296074 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:00.296132 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.296099 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-kube-api-access-vwhcz" (OuterVolumeSpecName: "kube-api-access-vwhcz") pod "af49c790-21af-4755-9d76-fc0c29d3da4c" (UID: "af49c790-21af-4755-9d76-fc0c29d3da4c"). InnerVolumeSpecName "kube-api-access-vwhcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:47:00.394971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.394929 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwhcz\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-kube-api-access-vwhcz\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.394971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.394964 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.394971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.394975 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.394971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.394983 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.395256 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.394991 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af49c790-21af-4755-9d76-fc0c29d3da4c-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:00.578611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.578517 2566 generic.go:356] "Generic (PLEG): container finished" podID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerID="7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293" exitCode=0 Jul 27 19:47:00.578611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.578578 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" event={"ID":"af49c790-21af-4755-9d76-fc0c29d3da4c","Type":"ContainerDied","Data":"7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293"} Jul 27 19:47:00.578611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.578607 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" Jul 27 19:47:00.578924 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.578640 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb" event={"ID":"af49c790-21af-4755-9d76-fc0c29d3da4c","Type":"ContainerDied","Data":"a689cb933dda2f8a3cce1e95283576275031bcac4e476fa8a75a16caa36992dd"} Jul 27 19:47:00.578924 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.578669 2566 scope.go:117] "RemoveContainer" containerID="7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293" Jul 27 19:47:00.588989 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.588968 2566 scope.go:117] "RemoveContainer" containerID="1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9" Jul 27 19:47:00.601210 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.600911 2566 scope.go:117] "RemoveContainer" containerID="7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293" Jul 27 19:47:00.601357 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:47:00.601334 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293\": container with ID starting with 7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293 not found: ID does not exist" containerID="7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293" Jul 27 19:47:00.601425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.601368 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293"} err="failed to get container status \"7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293\": rpc error: code = NotFound desc = could not find container \"7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293\": container with ID starting with 7dd52518034b30d566fe60350e20c6c28e185ce0d72021cd138dd597b9fa6293 not found: ID does not exist" Jul 27 19:47:00.601425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.601417 2566 scope.go:117] "RemoveContainer" containerID="1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9" Jul 27 19:47:00.601727 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:47:00.601698 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9\": container with ID starting with 1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9 not found: ID does not exist" containerID="1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9" Jul 27 19:47:00.601817 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.601735 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9"} err="failed to get container status \"1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9\": rpc error: code = NotFound desc = could not find container \"1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9\": container with ID starting with 1979b39e2707939739f4eee28de9d9c706a8eeb08f0cf54e6a72bde705f5e6b9 not found: ID does not exist" Jul 27 19:47:00.603826 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.603802 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb"] Jul 27 19:47:00.610504 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:00.610465 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-a312c841-gw-openshift-default-6664d64b6d-4bdwb"] Jul 27 19:47:01.546561 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:01.546512 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" path="/var/lib/kubelet/pods/af49c790-21af-4755-9d76-fc0c29d3da4c/volumes" Jul 27 19:47:07.662560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:07.662525 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:47:07.665693 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:07.665666 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:47:13.567273 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.567233 2566 patch_prober.go:28] interesting pod/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:47:13.567727 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.567300 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:47:13.682047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.682013 2566 generic.go:356] "Generic (PLEG): container finished" podID="21e8c485-44c7-4694-bd85-a06a26578f37" containerID="4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1" exitCode=0 Jul 27 19:47:13.682221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.682070 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" event={"ID":"21e8c485-44c7-4694-bd85-a06a26578f37","Type":"ContainerDied","Data":"4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1"} Jul 27 19:47:13.682523 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.682509 2566 scope.go:117] "RemoveContainer" containerID="4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1" Jul 27 19:47:13.683780 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.683719 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:13.683908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.683840 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:13.683908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:13.683882 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:14.689958 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:14.689925 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" event={"ID":"21e8c485-44c7-4694-bd85-a06a26578f37","Type":"ContainerStarted","Data":"00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05"} Jul 27 19:47:14.987030 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:14.986928 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:14.987270 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:14.987246 2566 patch_prober.go:28] interesting pod/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:47:14.987323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:14.987299 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:47:15.987636 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:15.987603 2566 patch_prober.go:28] interesting pod/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:47:15.988041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:15.987655 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:47:16.987265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:16.987222 2566 patch_prober.go:28] interesting pod/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:47:16.987457 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:16.987285 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:47:17.987147 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:17.987108 2566 patch_prober.go:28] interesting pod/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:47:17.987574 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:17.987170 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:47:18.987125 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:18.987086 2566 patch_prober.go:28] interesting pod/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Jul 27 19:47:18.987318 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:18.987147 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Jul 27 19:47:19.991072 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:19.991037 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:19.991587 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:19.991446 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:19.992258 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:19.992237 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:23.700740 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.700685 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2"] Jul 27 19:47:23.701343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.701267 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" Jul 27 19:47:23.701343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.701285 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" Jul 27 19:47:23.701343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.701298 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" Jul 27 19:47:23.701343 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.701304 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" Jul 27 19:47:23.701830 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.701411 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" Jul 27 19:47:23.701830 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.701421 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="af49c790-21af-4755-9d76-fc0c29d3da4c" containerName="istio-proxy" Jul 27 19:47:23.704589 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.704571 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:23.707292 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.707268 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:47:23.713329 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.713143 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2"] Jul 27 19:47:23.721627 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.721594 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a473ae1-0595-43bf-af44-27fd673ceb0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2\" (UID: \"7a473ae1-0595-43bf-af44-27fd673ceb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:23.823084 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.823046 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a473ae1-0595-43bf-af44-27fd673ceb0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2\" (UID: \"7a473ae1-0595-43bf-af44-27fd673ceb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:23.823847 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:23.823824 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a473ae1-0595-43bf-af44-27fd673ceb0a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2\" (UID: \"7a473ae1-0595-43bf-af44-27fd673ceb0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:24.018074 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:24.017980 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:24.181089 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:24.181060 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2"] Jul 27 19:47:24.183120 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:47:24.183090 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a473ae1_0595_43bf_af44_27fd673ceb0a.slice/crio-069d1b9bc1a0d1138d3a214cdd0cda150a36732999201fde1f2070e20314d5a6 WatchSource:0}: Error finding container 069d1b9bc1a0d1138d3a214cdd0cda150a36732999201fde1f2070e20314d5a6: Status 404 returned error can't find the container with id 069d1b9bc1a0d1138d3a214cdd0cda150a36732999201fde1f2070e20314d5a6 Jul 27 19:47:24.751646 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:24.751606 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" event={"ID":"7a473ae1-0595-43bf-af44-27fd673ceb0a","Type":"ContainerStarted","Data":"b5752b8742b7bf9d3a3bb1f83ee6a9b7a3482a6a4d4c110e54c5d4d46213197a"} Jul 27 19:47:24.752186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:24.751690 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" event={"ID":"7a473ae1-0595-43bf-af44-27fd673ceb0a","Type":"ContainerStarted","Data":"069d1b9bc1a0d1138d3a214cdd0cda150a36732999201fde1f2070e20314d5a6"} Jul 27 19:47:24.768206 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:24.768153 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" podStartSLOduration=1.768137257 podStartE2EDuration="1.768137257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:24.766471399 +0000 UTC m=+1817.760968375" watchObservedRunningTime="2026-07-27 19:47:24.768137257 +0000 UTC m=+1817.762634482" Jul 27 19:47:25.759410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:25.759371 2566 generic.go:356] "Generic (PLEG): container finished" podID="7a473ae1-0595-43bf-af44-27fd673ceb0a" containerID="b5752b8742b7bf9d3a3bb1f83ee6a9b7a3482a6a4d4c110e54c5d4d46213197a" exitCode=0 Jul 27 19:47:25.759849 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:25.759430 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" event={"ID":"7a473ae1-0595-43bf-af44-27fd673ceb0a","Type":"ContainerDied","Data":"b5752b8742b7bf9d3a3bb1f83ee6a9b7a3482a6a4d4c110e54c5d4d46213197a"} Jul 27 19:47:26.912260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:26.912227 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:26.955755 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:26.955711 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a473ae1-0595-43bf-af44-27fd673ceb0a-maas-api-service-ca\") pod \"7a473ae1-0595-43bf-af44-27fd673ceb0a\" (UID: \"7a473ae1-0595-43bf-af44-27fd673ceb0a\") " Jul 27 19:47:26.956079 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:26.956058 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a473ae1-0595-43bf-af44-27fd673ceb0a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7a473ae1-0595-43bf-af44-27fd673ceb0a" (UID: "7a473ae1-0595-43bf-af44-27fd673ceb0a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:47:26.956205 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:26.956184 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a473ae1-0595-43bf-af44-27fd673ceb0a-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:27.772741 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:27.772710 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" Jul 27 19:47:27.772924 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:27.772705 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2" event={"ID":"7a473ae1-0595-43bf-af44-27fd673ceb0a","Type":"ContainerDied","Data":"069d1b9bc1a0d1138d3a214cdd0cda150a36732999201fde1f2070e20314d5a6"} Jul 27 19:47:27.772924 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:27.772833 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="069d1b9bc1a0d1138d3a214cdd0cda150a36732999201fde1f2070e20314d5a6" Jul 27 19:47:38.980930 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:38.980887 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2"] Jul 27 19:47:38.985669 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:38.985638 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0b0189d4-99lm2"] Jul 27 19:47:39.546585 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:39.546549 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a473ae1-0595-43bf-af44-27fd673ceb0a" path="/var/lib/kubelet/pods/7a473ae1-0595-43bf-af44-27fd673ceb0a/volumes" Jul 27 19:47:40.158691 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.158651 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2"] Jul 27 19:47:40.159184 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.158929 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" containerID="cri-o://00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05" gracePeriod=30 Jul 27 19:47:40.868209 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.868167 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn"] Jul 27 19:47:40.868732 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.868713 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a473ae1-0595-43bf-af44-27fd673ceb0a" containerName="revoke-keys" Jul 27 19:47:40.868850 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.868735 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a473ae1-0595-43bf-af44-27fd673ceb0a" containerName="revoke-keys" Jul 27 19:47:40.868933 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.868919 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a473ae1-0595-43bf-af44-27fd673ceb0a" containerName="revoke-keys" Jul 27 19:47:40.872092 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.872068 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.875113 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.875090 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-67e1471d-openshift-default-dockercfg-j4c2t\"" Jul 27 19:47:40.905855 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.905815 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn"] Jul 27 19:47:40.992647 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992609 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-certs\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.992647 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992647 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d034241-54c4-46af-a58a-0ba43b567f94-istiod-ca-cert\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.992925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992792 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d034241-54c4-46af-a58a-0ba43b567f94-istio-podinfo\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.992925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992840 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49k4h\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-kube-api-access-49k4h\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.992925 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-envoy\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.993031 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992924 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-data\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.993031 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.992966 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-credential-socket\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.993031 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.993010 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-socket\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:40.993129 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:40.993030 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-istio-token\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094624 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094587 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d034241-54c4-46af-a58a-0ba43b567f94-istio-podinfo\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094636 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49k4h\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-kube-api-access-49k4h\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094674 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-envoy\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094710 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-data\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-credential-socket\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094829 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-socket\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.094863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094852 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-istio-token\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.095206 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094877 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-certs\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.095206 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.094908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d034241-54c4-46af-a58a-0ba43b567f94-istiod-ca-cert\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.095206 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.095185 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-data\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.095341 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.095223 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-socket\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.095605 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.095582 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-credential-socket\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.095605 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.095596 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-certs\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.096049 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.096020 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d034241-54c4-46af-a58a-0ba43b567f94-istiod-ca-cert\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.097177 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.097150 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-envoy\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.097416 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.097400 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d034241-54c4-46af-a58a-0ba43b567f94-istio-podinfo\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.172496 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.172409 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-istio-token\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.177312 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.177288 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49k4h\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-kube-api-access-49k4h\") pod \"e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.183442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.183412 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:41.398859 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.398830 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn"] Jul 27 19:47:41.404659 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.404616 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:41.404754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.404704 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:41.404754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.404739 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:41.859553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.859442 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" event={"ID":"8d034241-54c4-46af-a58a-0ba43b567f94","Type":"ContainerStarted","Data":"6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954"} Jul 27 19:47:41.859553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.859505 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" event={"ID":"8d034241-54c4-46af-a58a-0ba43b567f94","Type":"ContainerStarted","Data":"147d66aeaff0acb9b6c6739bfd28a69e2fe44df70874902c469cee1ce5237ad9"} Jul 27 19:47:41.916840 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:41.916788 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podStartSLOduration=1.91675073 podStartE2EDuration="1.91675073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:47:41.91310448 +0000 UTC m=+1834.907601455" watchObservedRunningTime="2026-07-27 19:47:41.91675073 +0000 UTC m=+1834.911247707" Jul 27 19:47:42.183751 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:42.183718 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:42.188901 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:42.188871 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:42.865101 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:42.865075 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:42.866144 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:42.866119 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:45.438503 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.438475 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:45.546143 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546115 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tjsz7\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-kube-api-access-tjsz7\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546143 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546148 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-istio-token\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546172 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-certs\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546197 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-data\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546236 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e8c485-44c7-4694-bd85-a06a26578f37-istiod-ca-cert\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546331 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-credential-socket\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546358 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-envoy\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546436 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-socket\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546483 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e8c485-44c7-4694-bd85-a06a26578f37-istio-podinfo\") pod \"21e8c485-44c7-4694-bd85-a06a26578f37\" (UID: \"21e8c485-44c7-4694-bd85-a06a26578f37\") " Jul 27 19:47:45.546657 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546526 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:45.546847 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546738 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:45.547019 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.546996 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.547085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.547026 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.547464 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.547364 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-data" (OuterVolumeSpecName: "istio-data") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:45.547464 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.547433 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:45.547734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.547709 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21e8c485-44c7-4694-bd85-a06a26578f37-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:47:45.548839 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.548818 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-istio-token" (OuterVolumeSpecName: "istio-token") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:47:45.549593 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.549563 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21e8c485-44c7-4694-bd85-a06a26578f37-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:47:45.549593 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.549582 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-kube-api-access-tjsz7" (OuterVolumeSpecName: "kube-api-access-tjsz7") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "kube-api-access-tjsz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:47:45.549747 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.549725 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "21e8c485-44c7-4694-bd85-a06a26578f37" (UID: "21e8c485-44c7-4694-bd85-a06a26578f37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:47:45.648470 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648426 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsz7\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-kube-api-access-tjsz7\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.648470 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648470 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21e8c485-44c7-4694-bd85-a06a26578f37-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.648470 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648486 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.648748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648500 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21e8c485-44c7-4694-bd85-a06a26578f37-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.648748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648513 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.648748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648527 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21e8c485-44c7-4694-bd85-a06a26578f37-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.648748 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.648539 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21e8c485-44c7-4694-bd85-a06a26578f37-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:47:45.884776 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.884667 2566 generic.go:356] "Generic (PLEG): container finished" podID="21e8c485-44c7-4694-bd85-a06a26578f37" containerID="00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05" exitCode=0 Jul 27 19:47:45.884776 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.884728 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" Jul 27 19:47:45.884776 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.884747 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" event={"ID":"21e8c485-44c7-4694-bd85-a06a26578f37","Type":"ContainerDied","Data":"00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05"} Jul 27 19:47:45.885023 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.884856 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2" event={"ID":"21e8c485-44c7-4694-bd85-a06a26578f37","Type":"ContainerDied","Data":"e790a9ea1295d6efeeb3515292c80047520946c1c31dad06bcfedd2e7f5eb5d2"} Jul 27 19:47:45.885023 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.884881 2566 scope.go:117] "RemoveContainer" containerID="00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05" Jul 27 19:47:45.895038 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.895011 2566 scope.go:117] "RemoveContainer" containerID="4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1" Jul 27 19:47:45.905085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.905064 2566 scope.go:117] "RemoveContainer" containerID="00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05" Jul 27 19:47:45.905449 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:47:45.905429 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05\": container with ID starting with 00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05 not found: ID does not exist" containerID="00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05" Jul 27 19:47:45.905501 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.905460 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05"} err="failed to get container status \"00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05\": rpc error: code = NotFound desc = could not find container \"00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05\": container with ID starting with 00093b8696936ad15c0799398f2ee1fc8f40de03e80d3f74510badad2b93fb05 not found: ID does not exist" Jul 27 19:47:45.905501 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.905480 2566 scope.go:117] "RemoveContainer" containerID="4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1" Jul 27 19:47:45.905738 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:47:45.905718 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1\": container with ID starting with 4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1 not found: ID does not exist" containerID="4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1" Jul 27 19:47:45.905831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.905748 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1"} err="failed to get container status \"4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1\": rpc error: code = NotFound desc = could not find container \"4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1\": container with ID starting with 4adc62972fac798d7f7db3c89bc25132dcc59cb88edb89114f806d56dab0b0f1 not found: ID does not exist" Jul 27 19:47:45.933199 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.933158 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2"] Jul 27 19:47:45.941245 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:45.941210 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0b0189d4-openshift-default-66897454b8-4gjr2"] Jul 27 19:47:47.547037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:47.547002 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" path="/var/lib/kubelet/pods/21e8c485-44c7-4694-bd85-a06a26578f37/volumes" Jul 27 19:47:56.963869 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:56.963781 2566 generic.go:356] "Generic (PLEG): container finished" podID="8d034241-54c4-46af-a58a-0ba43b567f94" containerID="6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954" exitCode=0 Jul 27 19:47:56.964281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:56.963859 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" event={"ID":"8d034241-54c4-46af-a58a-0ba43b567f94","Type":"ContainerDied","Data":"6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954"} Jul 27 19:47:56.964323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:56.964297 2566 scope.go:117] "RemoveContainer" containerID="6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954" Jul 27 19:47:56.965404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:56.965374 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:56.965478 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:56.965457 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:56.965515 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:56.965499 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:47:57.865654 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:57.865611 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:57.980413 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:57.980378 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" event={"ID":"8d034241-54c4-46af-a58a-0ba43b567f94","Type":"ContainerStarted","Data":"987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614"} Jul 27 19:47:58.183609 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:58.183581 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:47:58.184964 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:58.184936 2566 patch_prober.go:28] interesting pod/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 27 19:47:58.185096 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:58.184990 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 27 19:47:59.184005 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:59.183967 2566 patch_prober.go:28] interesting pod/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 27 19:47:59.184439 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:47:59.184023 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 27 19:48:00.184813 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:00.184753 2566 patch_prober.go:28] interesting pod/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 27 19:48:00.185232 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:00.184831 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 27 19:48:01.184655 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:01.184612 2566 patch_prober.go:28] interesting pod/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 27 19:48:01.184922 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:01.184683 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 27 19:48:02.184018 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:02.183979 2566 patch_prober.go:28] interesting pod/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 27 19:48:02.184210 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:02.184049 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 27 19:48:03.188351 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:03.188318 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:48:03.188802 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:03.188597 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:48:03.189245 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:03.189231 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:48:28.651664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:28.651624 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn"] Jul 27 19:48:28.652112 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:28.651888 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" containerID="cri-o://987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614" gracePeriod=30 Jul 27 19:48:33.191003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.190916 2566 patch_prober.go:28] interesting pod/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:48:33.191003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.190970 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:48:33.929273 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.929248 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:48:33.954574 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954541 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-data\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.954786 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954616 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49k4h\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-kube-api-access-49k4h\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.954786 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954650 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-credential-socket\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.954786 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954697 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-istio-token\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.954786 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954722 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d034241-54c4-46af-a58a-0ba43b567f94-istiod-ca-cert\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.954786 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954786 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-socket\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.955058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954862 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d034241-54c4-46af-a58a-0ba43b567f94-istio-podinfo\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.955058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954896 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-envoy\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.955058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.954936 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-certs\") pod \"8d034241-54c4-46af-a58a-0ba43b567f94\" (UID: \"8d034241-54c4-46af-a58a-0ba43b567f94\") " Jul 27 19:48:33.955214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.955086 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:48:33.955586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.955321 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:48:33.956024 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.956004 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:33.956166 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.956148 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:33.956257 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.956089 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:48:33.956340 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.956031 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d034241-54c4-46af-a58a-0ba43b567f94-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:48:33.956414 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.956346 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:48:33.957521 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.957485 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-kube-api-access-49k4h" (OuterVolumeSpecName: "kube-api-access-49k4h") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "kube-api-access-49k4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:48:33.958479 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.958455 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d034241-54c4-46af-a58a-0ba43b567f94-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:48:33.958627 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.958604 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:48:33.958715 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:33.958693 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d034241-54c4-46af-a58a-0ba43b567f94" (UID: "8d034241-54c4-46af-a58a-0ba43b567f94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:48:34.057543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057507 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d034241-54c4-46af-a58a-0ba43b567f94-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.057543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057539 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.057543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057552 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.057840 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057560 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d034241-54c4-46af-a58a-0ba43b567f94-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.057840 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057569 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49k4h\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-kube-api-access-49k4h\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.057840 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057579 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d034241-54c4-46af-a58a-0ba43b567f94-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.057840 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.057587 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d034241-54c4-46af-a58a-0ba43b567f94-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:48:34.191641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.191548 2566 generic.go:356] "Generic (PLEG): container finished" podID="8d034241-54c4-46af-a58a-0ba43b567f94" containerID="987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614" exitCode=0 Jul 27 19:48:34.191641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.191605 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" Jul 27 19:48:34.192159 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.191606 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" event={"ID":"8d034241-54c4-46af-a58a-0ba43b567f94","Type":"ContainerDied","Data":"987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614"} Jul 27 19:48:34.192159 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.191721 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn" event={"ID":"8d034241-54c4-46af-a58a-0ba43b567f94","Type":"ContainerDied","Data":"147d66aeaff0acb9b6c6739bfd28a69e2fe44df70874902c469cee1ce5237ad9"} Jul 27 19:48:34.192159 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.191751 2566 scope.go:117] "RemoveContainer" containerID="987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614" Jul 27 19:48:34.201682 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.201663 2566 scope.go:117] "RemoveContainer" containerID="6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954" Jul 27 19:48:34.212889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.212857 2566 scope.go:117] "RemoveContainer" containerID="987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614" Jul 27 19:48:34.213472 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:48:34.213441 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614\": container with ID starting with 987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614 not found: ID does not exist" containerID="987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614" Jul 27 19:48:34.213621 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.213487 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614"} err="failed to get container status \"987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614\": rpc error: code = NotFound desc = could not find container \"987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614\": container with ID starting with 987f031990b0228ee36696468ed030cb4a2f280721196d38ab25d11f06135614 not found: ID does not exist" Jul 27 19:48:34.213621 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.213512 2566 scope.go:117] "RemoveContainer" containerID="6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954" Jul 27 19:48:34.213874 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:48:34.213820 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954\": container with ID starting with 6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954 not found: ID does not exist" containerID="6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954" Jul 27 19:48:34.213874 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.213844 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954"} err="failed to get container status \"6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954\": rpc error: code = NotFound desc = could not find container \"6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954\": container with ID starting with 6b9d926a056fa79251e4ebb2cca3c1de856bad069d00682a487cf61377c5c954 not found: ID does not exist" Jul 27 19:48:34.215549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.215525 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn"] Jul 27 19:48:34.217822 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:34.217799 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-67e1471d-openshift-default-6b9485c664-gmvmn"] Jul 27 19:48:35.547502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:48:35.547456 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" path="/var/lib/kubelet/pods/8d034241-54c4-46af-a58a-0ba43b567f94/volumes" Jul 27 19:50:12.652170 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652128 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5"] Jul 27 19:50:12.652706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652596 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" Jul 27 19:50:12.652706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652607 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" Jul 27 19:50:12.652706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652614 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" Jul 27 19:50:12.652706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652621 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" Jul 27 19:50:12.652706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652630 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" Jul 27 19:50:12.652706 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652636 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" Jul 27 19:50:12.652944 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652716 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" Jul 27 19:50:12.652944 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.652726 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" Jul 27 19:50:12.656037 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.656013 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.658392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.658369 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-eae5981c-openshift-default-dockercfg-5kf8r\"" Jul 27 19:50:12.688049 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.688017 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5"] Jul 27 19:50:12.766313 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-data\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766313 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766319 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-certs\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766377 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istiod-ca-cert\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766484 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wrj\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-kube-api-access-98wrj\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766528 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766508 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-socket\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766549 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-envoy\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766581 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-podinfo\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766631 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766608 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-token\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.766737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.766632 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-credential-socket\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867304 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-data\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867366 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-certs\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867395 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istiod-ca-cert\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867481 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wrj\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-kube-api-access-98wrj\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867736 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867614 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-socket\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867736 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867693 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-envoy\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867888 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867737 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-podinfo\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867888 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867801 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-token\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867888 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867804 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-data\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867888 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867824 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-certs\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.867888 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.867841 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-credential-socket\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.868139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.868072 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-socket\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.868139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.868086 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-credential-socket\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.868251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.868233 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istiod-ca-cert\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.870130 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.870109 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-envoy\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.870242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.870231 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-podinfo\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.876884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.876847 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-token\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.877010 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.876917 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wrj\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-kube-api-access-98wrj\") pod \"e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:12.966555 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:12.966515 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:13.131189 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.131155 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5"] Jul 27 19:50:13.132859 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:50:13.132830 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e0dbaf_764c_4d0f_abd1_e47092dd4b9a.slice/crio-640f0f781a2a8edf3cf6f169a8f450176738361de95b093b15cd338314159316 WatchSource:0}: Error finding container 640f0f781a2a8edf3cf6f169a8f450176738361de95b093b15cd338314159316: Status 404 returned error can't find the container with id 640f0f781a2a8edf3cf6f169a8f450176738361de95b093b15cd338314159316 Jul 27 19:50:13.135126 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.135090 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:50:13.135241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.135160 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:50:13.135241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.135190 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:50:13.784070 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.784027 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" event={"ID":"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a","Type":"ContainerStarted","Data":"656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777"} Jul 27 19:50:13.784489 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.784092 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" event={"ID":"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a","Type":"ContainerStarted","Data":"640f0f781a2a8edf3cf6f169a8f450176738361de95b093b15cd338314159316"} Jul 27 19:50:13.811313 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.811250 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podStartSLOduration=1.811232819 podStartE2EDuration="1.811232819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:50:13.809236026 +0000 UTC m=+1986.803733014" watchObservedRunningTime="2026-07-27 19:50:13.811232819 +0000 UTC m=+1986.805729793" Jul 27 19:50:13.967089 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:13.967045 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:14.972542 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:14.972509 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:15.799032 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:15.799000 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:15.800143 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:15.800119 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:31.841022 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:31.840951 2566 patch_prober.go:28] interesting pod/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:50:31.841571 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:31.841045 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:50:32.908161 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:32.908076 2566 generic.go:356] "Generic (PLEG): container finished" podID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerID="656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777" exitCode=0 Jul 27 19:50:32.908161 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:32.908144 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" event={"ID":"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a","Type":"ContainerDied","Data":"656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777"} Jul 27 19:50:32.908649 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:32.908632 2566 scope.go:117] "RemoveContainer" containerID="656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777" Jul 27 19:50:32.909710 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:32.909673 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:50:32.909844 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:32.909822 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:50:32.909890 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:32.909870 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:50:33.920799 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:33.920731 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" event={"ID":"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a","Type":"ContainerStarted","Data":"6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc"} Jul 27 19:50:33.967717 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:33.967668 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:33.968086 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:33.968064 2566 patch_prober.go:28] interesting pod/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 27 19:50:33.968139 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:33.968111 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 27 19:50:34.967606 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:34.967563 2566 patch_prober.go:28] interesting pod/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 27 19:50:34.968067 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:34.967624 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 27 19:50:35.967857 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:35.967814 2566 patch_prober.go:28] interesting pod/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 27 19:50:35.968275 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:35.967876 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 27 19:50:36.967874 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:36.967833 2566 patch_prober.go:28] interesting pod/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 27 19:50:36.968246 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:36.967908 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 27 19:50:37.970692 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:37.970663 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:37.971195 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:37.971009 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:37.971804 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:37.971787 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:49.399967 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:49.399919 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5"] Jul 27 19:50:49.400564 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:49.400218 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" containerID="cri-o://6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc" gracePeriod=30 Jul 27 19:50:52.973331 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:52.973291 2566 patch_prober.go:28] interesting pod/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 27 19:50:52.973797 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:52.973353 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:50:54.668963 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.668889 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:54.710201 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710160 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98wrj\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-kube-api-access-98wrj\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710208 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istiod-ca-cert\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710359 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-data\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710534 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710441 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-envoy\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710534 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710476 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-socket\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710534 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710508 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-credential-socket\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710537 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-certs\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710576 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-token\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710608 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-podinfo\") pod \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\" (UID: \"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a\") " Jul 27 19:50:54.710696 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710628 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:50:54.710934 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710713 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:50:54.710934 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.710885 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:50:54.711047 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.711020 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.711140 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.711116 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:50:54.711221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.711185 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.711221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.711204 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.711363 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.711335 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-data" (OuterVolumeSpecName: "istio-data") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:50:54.712548 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.712525 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-kube-api-access-98wrj" (OuterVolumeSpecName: "kube-api-access-98wrj") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "kube-api-access-98wrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:50:54.712942 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.712921 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:50:54.713020 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.712964 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-token" (OuterVolumeSpecName: "istio-token") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:50:54.713190 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.713175 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" (UID: "47e0dbaf-764c-4d0f-abd1-e47092dd4b9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:50:54.811977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.811924 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.811977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.811969 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.811977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.811979 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.811977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.811988 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.811977 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.811998 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:54.812327 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:54.812007 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98wrj\" (UniqueName: \"kubernetes.io/projected/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a-kube-api-access-98wrj\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:50:55.053889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.053857 2566 generic.go:356] "Generic (PLEG): container finished" podID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerID="6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc" exitCode=0 Jul 27 19:50:55.054060 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.053903 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" event={"ID":"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a","Type":"ContainerDied","Data":"6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc"} Jul 27 19:50:55.054060 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.053922 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" Jul 27 19:50:55.054060 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.053944 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5" event={"ID":"47e0dbaf-764c-4d0f-abd1-e47092dd4b9a","Type":"ContainerDied","Data":"640f0f781a2a8edf3cf6f169a8f450176738361de95b093b15cd338314159316"} Jul 27 19:50:55.054060 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.053966 2566 scope.go:117] "RemoveContainer" containerID="6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc" Jul 27 19:50:55.063959 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.063941 2566 scope.go:117] "RemoveContainer" containerID="656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777" Jul 27 19:50:55.073003 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.072981 2566 scope.go:117] "RemoveContainer" containerID="6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc" Jul 27 19:50:55.073313 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:50:55.073295 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc\": container with ID starting with 6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc not found: ID does not exist" containerID="6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc" Jul 27 19:50:55.073358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.073325 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc"} err="failed to get container status \"6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc\": rpc error: code = NotFound desc = could not find container \"6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc\": container with ID starting with 6fe902ce3507ad2950e154ff763a04b75264fcbec6e3a81d4679feb17a1a97bc not found: ID does not exist" Jul 27 19:50:55.073358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.073344 2566 scope.go:117] "RemoveContainer" containerID="656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777" Jul 27 19:50:55.073637 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:50:55.073612 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777\": container with ID starting with 656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777 not found: ID does not exist" containerID="656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777" Jul 27 19:50:55.073754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.073643 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777"} err="failed to get container status \"656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777\": rpc error: code = NotFound desc = could not find container \"656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777\": container with ID starting with 656974bc69f6d6768acbc9e25fc69f9c21fe77473e172d7e548a296edcd72777 not found: ID does not exist" Jul 27 19:50:55.076671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.076646 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5"] Jul 27 19:50:55.084393 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.084368 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-eae5981c-openshift-default-5f7ff77b68-m8jn5"] Jul 27 19:50:55.546659 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:50:55.546628 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" path="/var/lib/kubelet/pods/47e0dbaf-764c-4d0f-abd1-e47092dd4b9a/volumes" Jul 27 19:51:09.594394 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594357 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq"] Jul 27 19:51:09.594852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594838 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" Jul 27 19:51:09.594852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594853 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" Jul 27 19:51:09.594932 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594863 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" Jul 27 19:51:09.594932 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594869 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" Jul 27 19:51:09.594932 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594885 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" Jul 27 19:51:09.594932 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594891 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" Jul 27 19:51:09.595059 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594954 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" Jul 27 19:51:09.595059 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594964 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d034241-54c4-46af-a58a-0ba43b567f94" containerName="istio-proxy" Jul 27 19:51:09.595059 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.594972 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="21e8c485-44c7-4694-bd85-a06a26578f37" containerName="istio-proxy" Jul 27 19:51:09.599917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.599893 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.602488 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.602458 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-ef75af22-openshift-default-dockercfg-t267x\"" Jul 27 19:51:09.612826 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.612791 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq"] Jul 27 19:51:09.668410 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668357 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzdx\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-kube-api-access-dtzdx\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668462 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-credential-socket\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668497 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644962dc-1e76-4d4f-b72c-7021f2e16f46-istiod-ca-cert\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668534 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-data\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668561 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-token\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-certs\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668616 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-podinfo\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668683 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-socket\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.668788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.668699 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-envoy\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.769726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769677 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-credential-socket\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.769726 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769727 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644962dc-1e76-4d4f-b72c-7021f2e16f46-istiod-ca-cert\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769755 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-data\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769831 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-token\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769863 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-certs\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769897 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-podinfo\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769971 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-socket\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.769993 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-envoy\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770357 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.770059 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzdx\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-kube-api-access-dtzdx\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770357 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.770299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-certs\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.770351 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-data\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.770408 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-credential-socket\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770458 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.770416 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-socket\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.770569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.770534 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644962dc-1e76-4d4f-b72c-7021f2e16f46-istiod-ca-cert\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.772413 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.772389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-envoy\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.772549 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.772529 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-podinfo\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.778220 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.778189 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-token\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.778350 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.778330 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzdx\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-kube-api-access-dtzdx\") pod \"e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:09.912875 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:09.912753 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:10.079947 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:10.079920 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq"] Jul 27 19:51:10.082184 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:51:10.082141 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644962dc_1e76_4d4f_b72c_7021f2e16f46.slice/crio-4a8cbc01a2f1a26e4f66daa3810c5694e1d3b2916d8fe5303788662d0fba2ad9 WatchSource:0}: Error finding container 4a8cbc01a2f1a26e4f66daa3810c5694e1d3b2916d8fe5303788662d0fba2ad9: Status 404 returned error can't find the container with id 4a8cbc01a2f1a26e4f66daa3810c5694e1d3b2916d8fe5303788662d0fba2ad9 Jul 27 19:51:10.085262 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:10.085185 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:51:10.085453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:10.085292 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:51:10.085453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:10.085332 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:51:10.142480 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:10.142448 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" event={"ID":"644962dc-1e76-4d4f-b72c-7021f2e16f46","Type":"ContainerStarted","Data":"4a8cbc01a2f1a26e4f66daa3810c5694e1d3b2916d8fe5303788662d0fba2ad9"} Jul 27 19:51:11.149214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:11.149176 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" event={"ID":"644962dc-1e76-4d4f-b72c-7021f2e16f46","Type":"ContainerStarted","Data":"006e802d5955dc7aa93ef47c90057a4df9e94fddee3631b3744d19400da88192"} Jul 27 19:51:11.172000 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:11.171950 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podStartSLOduration=2.171931814 podStartE2EDuration="2.171931814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:51:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:51:11.17049988 +0000 UTC m=+2044.164996857" watchObservedRunningTime="2026-07-27 19:51:11.171931814 +0000 UTC m=+2044.166428860" Jul 27 19:51:11.913135 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:11.913098 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:11.919302 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:11.919273 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:12.154894 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:12.154861 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:12.155741 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:12.155722 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:27.156976 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.156938 2566 patch_prober.go:28] interesting pod/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:51:27.157404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.156998 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:51:27.248962 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.248927 2566 generic.go:356] "Generic (PLEG): container finished" podID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerID="006e802d5955dc7aa93ef47c90057a4df9e94fddee3631b3744d19400da88192" exitCode=0 Jul 27 19:51:27.249149 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.248998 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" event={"ID":"644962dc-1e76-4d4f-b72c-7021f2e16f46","Type":"ContainerDied","Data":"006e802d5955dc7aa93ef47c90057a4df9e94fddee3631b3744d19400da88192"} Jul 27 19:51:27.249472 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.249456 2566 scope.go:117] "RemoveContainer" containerID="006e802d5955dc7aa93ef47c90057a4df9e94fddee3631b3744d19400da88192" Jul 27 19:51:27.250664 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.250634 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:51:27.250727 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.250712 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:51:27.250788 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:27.250748 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:51:28.256987 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:28.256956 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" event={"ID":"644962dc-1e76-4d4f-b72c-7021f2e16f46","Type":"ContainerStarted","Data":"601b37f448c197ab75102cfe413f5bd6417e96284a7ea37d709372dcc8670865"} Jul 27 19:51:28.913459 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:28.913365 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:28.913892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:28.913864 2566 patch_prober.go:28] interesting pod/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:51:28.914097 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:28.914070 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:51:29.914027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:29.913986 2566 patch_prober.go:28] interesting pod/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:51:29.914658 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:29.914624 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:51:30.913889 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:30.913844 2566 patch_prober.go:28] interesting pod/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:51:30.914111 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:30.913920 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:51:31.914044 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:31.913996 2566 patch_prober.go:28] interesting pod/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Jul 27 19:51:31.914265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:31.914076 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Jul 27 19:51:32.917178 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:32.917150 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:32.917659 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:32.917646 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:51:32.918369 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:51:32.918351 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:52:07.133966 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.133919 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz"] Jul 27 19:52:07.134814 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.134746 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e0dbaf-764c-4d0f-abd1-e47092dd4b9a" containerName="istio-proxy" Jul 27 19:52:07.138194 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.138175 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:07.141174 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.141140 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:52:07.149711 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.149683 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz"] Jul 27 19:52:07.169681 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.169644 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efce83c-887f-4146-afdb-fe251d2abecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz\" (UID: \"5efce83c-887f-4146-afdb-fe251d2abecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:07.271038 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.271003 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efce83c-887f-4146-afdb-fe251d2abecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz\" (UID: \"5efce83c-887f-4146-afdb-fe251d2abecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:07.271721 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.271696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efce83c-887f-4146-afdb-fe251d2abecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz\" (UID: \"5efce83c-887f-4146-afdb-fe251d2abecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:07.452161 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.452126 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:52:07.460262 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.460230 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:07.613242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.613201 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz"] Jul 27 19:52:07.614988 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:52:07.614954 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5efce83c_887f_4146_afdb_fe251d2abecf.slice/crio-970b5836460764def4b7ac787c1ac0f06729d270ebf2bd73b23c78081eebf542 WatchSource:0}: Error finding container 970b5836460764def4b7ac787c1ac0f06729d270ebf2bd73b23c78081eebf542: Status 404 returned error can't find the container with id 970b5836460764def4b7ac787c1ac0f06729d270ebf2bd73b23c78081eebf542 Jul 27 19:52:07.710457 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.710353 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:52:07.711730 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:07.711704 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:52:08.499308 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:08.499269 2566 generic.go:356] "Generic (PLEG): container finished" podID="5efce83c-887f-4146-afdb-fe251d2abecf" containerID="70dab5d6746596fb9fdb7675cca4ef4c67afab6d58fad6dedf87027c52044694" exitCode=0 Jul 27 19:52:08.499710 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:08.499349 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" event={"ID":"5efce83c-887f-4146-afdb-fe251d2abecf","Type":"ContainerDied","Data":"70dab5d6746596fb9fdb7675cca4ef4c67afab6d58fad6dedf87027c52044694"} Jul 27 19:52:08.499710 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:08.499435 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" event={"ID":"5efce83c-887f-4146-afdb-fe251d2abecf","Type":"ContainerStarted","Data":"970b5836460764def4b7ac787c1ac0f06729d270ebf2bd73b23c78081eebf542"} Jul 27 19:52:09.656325 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:09.656297 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:09.693829 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:09.693789 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efce83c-887f-4146-afdb-fe251d2abecf-maas-api-service-ca\") pod \"5efce83c-887f-4146-afdb-fe251d2abecf\" (UID: \"5efce83c-887f-4146-afdb-fe251d2abecf\") " Jul 27 19:52:09.694151 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:09.694128 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5efce83c-887f-4146-afdb-fe251d2abecf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5efce83c-887f-4146-afdb-fe251d2abecf" (UID: "5efce83c-887f-4146-afdb-fe251d2abecf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:52:09.795437 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:09.795343 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5efce83c-887f-4146-afdb-fe251d2abecf-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:52:10.513201 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:10.513164 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" event={"ID":"5efce83c-887f-4146-afdb-fe251d2abecf","Type":"ContainerDied","Data":"970b5836460764def4b7ac787c1ac0f06729d270ebf2bd73b23c78081eebf542"} Jul 27 19:52:10.513433 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:10.513223 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="970b5836460764def4b7ac787c1ac0f06729d270ebf2bd73b23c78081eebf542" Jul 27 19:52:10.513433 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:10.513224 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz" Jul 27 19:52:27.248995 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:27.248948 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz"] Jul 27 19:52:27.252450 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:27.252417 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ef75af22-m2hxz"] Jul 27 19:52:27.554577 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:52:27.554476 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5efce83c-887f-4146-afdb-fe251d2abecf" path="/var/lib/kubelet/pods/5efce83c-887f-4146-afdb-fe251d2abecf/volumes" Jul 27 19:53:07.823568 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:07.823531 2566 scope.go:117] "RemoveContainer" containerID="56c1738bfd3e4c7909c00f9f3807092d373878e31290cf3576b1b8ba7ba355d8" Jul 27 19:53:25.796898 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:25.796862 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq"] Jul 27 19:53:25.797405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:25.797070 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" containerID="cri-o://601b37f448c197ab75102cfe413f5bd6417e96284a7ea37d709372dcc8670865" gracePeriod=30 Jul 27 19:53:30.996246 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:30.996197 2566 generic.go:356] "Generic (PLEG): container finished" podID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerID="601b37f448c197ab75102cfe413f5bd6417e96284a7ea37d709372dcc8670865" exitCode=0 Jul 27 19:53:30.996833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:30.996422 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" event={"ID":"644962dc-1e76-4d4f-b72c-7021f2e16f46","Type":"ContainerDied","Data":"601b37f448c197ab75102cfe413f5bd6417e96284a7ea37d709372dcc8670865"} Jul 27 19:53:30.996833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:30.996489 2566 scope.go:117] "RemoveContainer" containerID="006e802d5955dc7aa93ef47c90057a4df9e94fddee3631b3744d19400da88192" Jul 27 19:53:31.067025 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.066937 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:53:31.096994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.096957 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-podinfo\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097023 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-credential-socket\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097049 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-token\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097104 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-data\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097144 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-envoy\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097208 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-certs\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097267 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dtzdx\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-kube-api-access-dtzdx\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097305 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-socket\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097336 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644962dc-1e76-4d4f-b72c-7021f2e16f46-istiod-ca-cert\") pod \"644962dc-1e76-4d4f-b72c-7021f2e16f46\" (UID: \"644962dc-1e76-4d4f-b72c-7021f2e16f46\") " Jul 27 19:53:31.097599 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097526 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:53:31.097960 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097772 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:53:31.097960 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.097874 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644962dc-1e76-4d4f-b72c-7021f2e16f46-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:53:31.098379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.098212 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:53:31.098379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.098339 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.098379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.098363 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.098379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.098378 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/644962dc-1e76-4d4f-b72c-7021f2e16f46-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.098591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.098390 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.098591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.098340 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-data" (OuterVolumeSpecName: "istio-data") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:53:31.099556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.099520 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:53:31.099905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.099883 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:53:31.100245 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.100224 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-token" (OuterVolumeSpecName: "istio-token") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:53:31.101170 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.101144 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-kube-api-access-dtzdx" (OuterVolumeSpecName: "kube-api-access-dtzdx") pod "644962dc-1e76-4d4f-b72c-7021f2e16f46" (UID: "644962dc-1e76-4d4f-b72c-7021f2e16f46"). InnerVolumeSpecName "kube-api-access-dtzdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:53:31.199920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.199877 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzdx\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-kube-api-access-dtzdx\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.199920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.199910 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.199920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.199918 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.199920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.199927 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:31.200189 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:31.199937 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/644962dc-1e76-4d4f-b72c-7021f2e16f46-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:53:32.003905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:32.003873 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" Jul 27 19:53:32.004386 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:32.003906 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq" event={"ID":"644962dc-1e76-4d4f-b72c-7021f2e16f46","Type":"ContainerDied","Data":"4a8cbc01a2f1a26e4f66daa3810c5694e1d3b2916d8fe5303788662d0fba2ad9"} Jul 27 19:53:32.004386 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:32.003964 2566 scope.go:117] "RemoveContainer" containerID="601b37f448c197ab75102cfe413f5bd6417e96284a7ea37d709372dcc8670865" Jul 27 19:53:32.022663 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:32.022630 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq"] Jul 27 19:53:32.026395 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:32.026354 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-ef75af22-openshift-default-7dbfd4c4cb-jjvvq"] Jul 27 19:53:33.546543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:53:33.546506 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" path="/var/lib/kubelet/pods/644962dc-1e76-4d4f-b72c-7021f2e16f46/volumes" Jul 27 19:54:07.847852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:07.847818 2566 scope.go:117] "RemoveContainer" containerID="b5752b8742b7bf9d3a3bb1f83ee6a9b7a3482a6a4d4c110e54c5d4d46213197a" Jul 27 19:54:22.566701 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.566659 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f"] Jul 27 19:54:22.567164 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567152 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" Jul 27 19:54:22.567164 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567164 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" Jul 27 19:54:22.567239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567183 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" Jul 27 19:54:22.567239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567189 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" Jul 27 19:54:22.567239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567198 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5efce83c-887f-4146-afdb-fe251d2abecf" containerName="revoke-keys" Jul 27 19:54:22.567239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567203 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5efce83c-887f-4146-afdb-fe251d2abecf" containerName="revoke-keys" Jul 27 19:54:22.567362 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567293 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" Jul 27 19:54:22.567362 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.567304 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5efce83c-887f-4146-afdb-fe251d2abecf" containerName="revoke-keys" Jul 27 19:54:22.570625 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.570606 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.573509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.573481 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7bbd07-openshift-default-dockercfg-j4qwq\"" Jul 27 19:54:22.587533 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.587493 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f"] Jul 27 19:54:22.723318 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723273 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-envoy\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723536 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723349 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-socket\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723536 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723384 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-data\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723536 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723436 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-certs\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723536 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723506 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrf5\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-kube-api-access-jvrf5\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723569 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-credential-socket\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723607 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-podinfo\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723640 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-token\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.723709 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.723661 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c51423-533f-47c4-981b-37ad3e48aefe-istiod-ca-cert\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825277 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825176 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-credential-socket\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825277 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825235 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-podinfo\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825285 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-token\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825310 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c51423-533f-47c4-981b-37ad3e48aefe-istiod-ca-cert\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825372 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-envoy\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825437 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-socket\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825525 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825470 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-data\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825824 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825531 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-certs\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825824 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825564 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrf5\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-kube-api-access-jvrf5\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825824 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825644 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-credential-socket\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.825999 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.825876 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-socket\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.826041 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.826010 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-data\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.826099 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.826077 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-certs\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.826226 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.826203 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c51423-533f-47c4-981b-37ad3e48aefe-istiod-ca-cert\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.827666 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.827646 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-envoy\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.827956 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.827938 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-podinfo\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.842040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.842008 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrf5\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-kube-api-access-jvrf5\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.849588 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.849519 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-token\") pod \"e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:22.881919 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:22.881874 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:23.054999 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.054955 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f"] Jul 27 19:54:23.055728 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:54:23.055692 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1c51423_533f_47c4_981b_37ad3e48aefe.slice/crio-e10f90a7c5dc13e9a65fa5f023dc61b9d5fdb8d54f10992584399f0a77b0e719 WatchSource:0}: Error finding container e10f90a7c5dc13e9a65fa5f023dc61b9d5fdb8d54f10992584399f0a77b0e719: Status 404 returned error can't find the container with id e10f90a7c5dc13e9a65fa5f023dc61b9d5fdb8d54f10992584399f0a77b0e719 Jul 27 19:54:23.058143 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.058103 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:23.058215 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.058177 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:23.058260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.058213 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:23.309058 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.309017 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" event={"ID":"f1c51423-533f-47c4-981b-37ad3e48aefe","Type":"ContainerStarted","Data":"f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2"} Jul 27 19:54:23.309241 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.309121 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" event={"ID":"f1c51423-533f-47c4-981b-37ad3e48aefe","Type":"ContainerStarted","Data":"e10f90a7c5dc13e9a65fa5f023dc61b9d5fdb8d54f10992584399f0a77b0e719"} Jul 27 19:54:23.331926 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.331814 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podStartSLOduration=1.331795684 podStartE2EDuration="1.331795684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:54:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:54:23.329751034 +0000 UTC m=+2236.324248010" watchObservedRunningTime="2026-07-27 19:54:23.331795684 +0000 UTC m=+2236.326292669" Jul 27 19:54:23.882211 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:23.882170 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:24.887174 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:24.887143 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:25.325045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:25.325014 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:25.326027 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:25.326000 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:30.345884 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.345791 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg"] Jul 27 19:54:30.346720 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.346672 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="644962dc-1e76-4d4f-b72c-7021f2e16f46" containerName="istio-proxy" Jul 27 19:54:30.350316 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.350291 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.352990 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.352963 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-4ce65d-openshift-default-dockercfg-5hrfw\"" Jul 27 19:54:30.365732 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.365699 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg"] Jul 27 19:54:30.504733 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.504694 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-certs\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.504920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.504816 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54vlm\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-kube-api-access-54vlm\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.504920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.504870 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-socket\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.504920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.504886 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-envoy\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.504920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.504902 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-token\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.505080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.504997 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-credential-socket\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.505080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.505032 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istiod-ca-cert\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.505080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.505052 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-data\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.505080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.505069 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-podinfo\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.605975 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-certs\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606071 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54vlm\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-kube-api-access-54vlm\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606137 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-socket\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606161 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-envoy\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606184 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-token\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606244 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-credential-socket\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606263 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istiod-ca-cert\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606282 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-data\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606338 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606307 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-podinfo\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606694 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606448 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-certs\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606694 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606629 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-credential-socket\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606796 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606690 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-data\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.606951 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.606929 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-socket\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.607085 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.607007 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istiod-ca-cert\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.608524 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.608501 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-envoy\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.608751 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.608733 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-podinfo\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.617591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.617552 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-token\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.617755 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.617649 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54vlm\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-kube-api-access-54vlm\") pod \"e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.663532 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.663484 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:30.825694 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.825661 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg"] Jul 27 19:54:30.828971 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:54:30.828940 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba25fbf_38f8_4d05_9560_621b6a5e94e0.slice/crio-a6582e762069f94c6766ba6d98e03d391b151ceec9574d8235c28139118461a2 WatchSource:0}: Error finding container a6582e762069f94c6766ba6d98e03d391b151ceec9574d8235c28139118461a2: Status 404 returned error can't find the container with id a6582e762069f94c6766ba6d98e03d391b151ceec9574d8235c28139118461a2 Jul 27 19:54:30.831281 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.831243 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:30.831402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.831329 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:30.831402 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:30.831376 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:31.363024 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:31.362985 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" event={"ID":"6ba25fbf-38f8-4d05-9560-621b6a5e94e0","Type":"ContainerStarted","Data":"acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0"} Jul 27 19:54:31.363534 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:31.363038 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" event={"ID":"6ba25fbf-38f8-4d05-9560-621b6a5e94e0","Type":"ContainerStarted","Data":"a6582e762069f94c6766ba6d98e03d391b151ceec9574d8235c28139118461a2"} Jul 27 19:54:31.403374 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:31.403314 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podStartSLOduration=1.403297227 podStartE2EDuration="1.403297227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:54:31.402030399 +0000 UTC m=+2244.396527375" watchObservedRunningTime="2026-07-27 19:54:31.403297227 +0000 UTC m=+2244.397794254" Jul 27 19:54:31.664564 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:31.664472 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:31.666119 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:31.666092 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 27 19:54:31.666238 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:31.666146 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 27 19:54:32.668454 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:32.668420 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:33.377633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:33.377606 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:33.378724 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:33.378703 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:39.413662 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:39.413617 2566 generic.go:356] "Generic (PLEG): container finished" podID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerID="f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2" exitCode=0 Jul 27 19:54:39.414223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:39.413668 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" event={"ID":"f1c51423-533f-47c4-981b-37ad3e48aefe","Type":"ContainerDied","Data":"f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2"} Jul 27 19:54:39.414223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:39.414126 2566 scope.go:117] "RemoveContainer" containerID="f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2" Jul 27 19:54:39.415303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:39.415272 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:39.415382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:39.415344 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:39.415422 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:39.415381 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:40.325122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:40.325071 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:40.429036 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:40.428992 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" event={"ID":"f1c51423-533f-47c4-981b-37ad3e48aefe","Type":"ContainerStarted","Data":"94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc"} Jul 27 19:54:40.882079 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:40.882038 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:40.883437 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:40.883398 2566 patch_prober.go:28] interesting pod/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:54:40.883557 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:40.883462 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:54:41.882303 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:41.882259 2566 patch_prober.go:28] interesting pod/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:54:41.882728 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:41.882315 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:54:42.883257 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:42.883205 2566 patch_prober.go:28] interesting pod/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:54:42.883670 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:42.883272 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:54:43.882273 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:43.882229 2566 patch_prober.go:28] interesting pod/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Jul 27 19:54:43.882467 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:43.882294 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Jul 27 19:54:44.887604 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:44.887569 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:44.888117 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:44.887892 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:44.888710 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:44.888690 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:54:47.485233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:47.485202 2566 generic.go:356] "Generic (PLEG): container finished" podID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerID="acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0" exitCode=0 Jul 27 19:54:47.485695 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:47.485271 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" event={"ID":"6ba25fbf-38f8-4d05-9560-621b6a5e94e0","Type":"ContainerDied","Data":"acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0"} Jul 27 19:54:47.485785 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:47.485745 2566 scope.go:117] "RemoveContainer" containerID="acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0" Jul 27 19:54:47.486997 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:47.486959 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:47.487083 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:47.487036 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:47.487083 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:47.487067 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:54:48.378330 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:48.378293 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:48.493160 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:48.493128 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" event={"ID":"6ba25fbf-38f8-4d05-9560-621b6a5e94e0","Type":"ContainerStarted","Data":"e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3"} Jul 27 19:54:48.664371 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:48.664275 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:48.665944 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:48.665916 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 27 19:54:48.666089 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:48.665966 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 27 19:54:49.664280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:49.664245 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 27 19:54:49.664718 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:49.664303 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 27 19:54:50.664497 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:50.664463 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 27 19:54:50.665004 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:50.664533 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 27 19:54:51.664170 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:51.664134 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 27 19:54:51.664386 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:51.664192 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 27 19:54:52.664829 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:52.664751 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Jul 27 19:54:52.665251 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:52.664847 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Jul 27 19:54:53.668457 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:53.668423 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:53.668905 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:53.668687 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:54:53.669463 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:54:53.669442 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:56:03.762722 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:03.762633 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj"] Jul 27 19:56:03.766469 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:03.766446 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:03.772755 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:03.772721 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 19:56:03.795613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:03.795574 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj"] Jul 27 19:56:03.925346 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:03.925306 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4e5451-4900-4e8c-b813-415bd290adf8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj\" (UID: \"0f4e5451-4900-4e8c-b813-415bd290adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:04.027112 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.027009 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4e5451-4900-4e8c-b813-415bd290adf8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj\" (UID: \"0f4e5451-4900-4e8c-b813-415bd290adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:04.027683 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.027662 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4e5451-4900-4e8c-b813-415bd290adf8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj\" (UID: \"0f4e5451-4900-4e8c-b813-415bd290adf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:04.077906 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.077875 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:04.236450 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.236421 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj"] Jul 27 19:56:04.237593 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:56:04.237563 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4e5451_4900_4e8c_b813_415bd290adf8.slice/crio-e7bc828cca12f253bfdf8ef3f68eb8ba3fe74aba120ef506bff86839994e7fed WatchSource:0}: Error finding container e7bc828cca12f253bfdf8ef3f68eb8ba3fe74aba120ef506bff86839994e7fed: Status 404 returned error can't find the container with id e7bc828cca12f253bfdf8ef3f68eb8ba3fe74aba120ef506bff86839994e7fed Jul 27 19:56:04.938243 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.938200 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" event={"ID":"0f4e5451-4900-4e8c-b813-415bd290adf8","Type":"ContainerStarted","Data":"b5f2e594254e7ded40408cde0ce155229c014fc450b033c7afec54db6ccfc64c"} Jul 27 19:56:04.938645 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.938266 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" event={"ID":"0f4e5451-4900-4e8c-b813-415bd290adf8","Type":"ContainerStarted","Data":"e7bc828cca12f253bfdf8ef3f68eb8ba3fe74aba120ef506bff86839994e7fed"} Jul 27 19:56:04.957231 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:04.957175 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" podStartSLOduration=1.957159463 podStartE2EDuration="1.957159463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:56:04.956205497 +0000 UTC m=+2337.950702473" watchObservedRunningTime="2026-07-27 19:56:04.957159463 +0000 UTC m=+2337.951656437" Jul 27 19:56:05.945380 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:05.945341 2566 generic.go:356] "Generic (PLEG): container finished" podID="0f4e5451-4900-4e8c-b813-415bd290adf8" containerID="b5f2e594254e7ded40408cde0ce155229c014fc450b033c7afec54db6ccfc64c" exitCode=0 Jul 27 19:56:05.945842 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:05.945436 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" event={"ID":"0f4e5451-4900-4e8c-b813-415bd290adf8","Type":"ContainerDied","Data":"b5f2e594254e7ded40408cde0ce155229c014fc450b033c7afec54db6ccfc64c"} Jul 27 19:56:07.100369 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.100338 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:07.257983 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.257887 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4e5451-4900-4e8c-b813-415bd290adf8-maas-api-service-ca\") pod \"0f4e5451-4900-4e8c-b813-415bd290adf8\" (UID: \"0f4e5451-4900-4e8c-b813-415bd290adf8\") " Jul 27 19:56:07.258260 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.258237 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4e5451-4900-4e8c-b813-415bd290adf8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f4e5451-4900-4e8c-b813-415bd290adf8" (UID: "0f4e5451-4900-4e8c-b813-415bd290adf8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:56:07.359319 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.359275 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f4e5451-4900-4e8c-b813-415bd290adf8-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:07.957832 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.957801 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" Jul 27 19:56:07.957832 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.957817 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj" event={"ID":"0f4e5451-4900-4e8c-b813-415bd290adf8","Type":"ContainerDied","Data":"e7bc828cca12f253bfdf8ef3f68eb8ba3fe74aba120ef506bff86839994e7fed"} Jul 27 19:56:07.958048 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:07.957864 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7bc828cca12f253bfdf8ef3f68eb8ba3fe74aba120ef506bff86839994e7fed" Jul 27 19:56:18.817495 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:18.817451 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj"] Jul 27 19:56:18.828325 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:18.828290 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7bbd07-5j7tj"] Jul 27 19:56:19.547372 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:19.547337 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4e5451-4900-4e8c-b813-415bd290adf8" path="/var/lib/kubelet/pods/0f4e5451-4900-4e8c-b813-415bd290adf8/volumes" Jul 27 19:56:25.610734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:25.610688 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f"] Jul 27 19:56:25.611235 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:25.611021 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" containerID="cri-o://94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc" gracePeriod=30 Jul 27 19:56:29.891145 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:29.891106 2566 patch_prober.go:28] interesting pod/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:56:29.891560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:29.891177 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:56:30.895943 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:30.895919 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:56:30.999866 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:30.999832 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-podinfo\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000050 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:30.999890 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvrf5\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-kube-api-access-jvrf5\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000050 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:30.999930 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-envoy\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000127 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000045 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-data\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000127 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000086 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-token\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000127 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000120 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-certs\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000160 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c51423-533f-47c4-981b-37ad3e48aefe-istiod-ca-cert\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000265 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000254 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-socket\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000370 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000302 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-credential-socket\") pod \"f1c51423-533f-47c4-981b-37ad3e48aefe\" (UID: \"f1c51423-533f-47c4-981b-37ad3e48aefe\") " Jul 27 19:56:31.000581 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000536 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:31.000724 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000651 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1c51423-533f-47c4-981b-37ad3e48aefe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:56:31.000830 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000784 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:31.000893 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000860 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.000893 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:31.000893 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000882 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1c51423-533f-47c4-981b-37ad3e48aefe-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.001063 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.000916 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-data" (OuterVolumeSpecName: "istio-data") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:31.002358 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.002323 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:56:31.002453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.002363 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-kube-api-access-jvrf5" (OuterVolumeSpecName: "kube-api-access-jvrf5") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "kube-api-access-jvrf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:56:31.002453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.002413 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-token" (OuterVolumeSpecName: "istio-token") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:56:31.002453 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.002418 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f1c51423-533f-47c4-981b-37ad3e48aefe" (UID: "f1c51423-533f-47c4-981b-37ad3e48aefe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:31.094593 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.094556 2566 generic.go:356] "Generic (PLEG): container finished" podID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerID="94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc" exitCode=0 Jul 27 19:56:31.094793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.094618 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" Jul 27 19:56:31.094793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.094624 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" event={"ID":"f1c51423-533f-47c4-981b-37ad3e48aefe","Type":"ContainerDied","Data":"94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc"} Jul 27 19:56:31.094793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.094667 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f" event={"ID":"f1c51423-533f-47c4-981b-37ad3e48aefe","Type":"ContainerDied","Data":"e10f90a7c5dc13e9a65fa5f023dc61b9d5fdb8d54f10992584399f0a77b0e719"} Jul 27 19:56:31.094793 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.094691 2566 scope.go:117] "RemoveContainer" containerID="94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc" Jul 27 19:56:31.101594 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101565 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.101594 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101592 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.101812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101605 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvrf5\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-kube-api-access-jvrf5\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.101812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101615 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.101812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101625 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.101812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101633 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1c51423-533f-47c4-981b-37ad3e48aefe-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.101812 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.101640 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1c51423-533f-47c4-981b-37ad3e48aefe-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:31.107633 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.106783 2566 scope.go:117] "RemoveContainer" containerID="f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2" Jul 27 19:56:31.117651 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.117624 2566 scope.go:117] "RemoveContainer" containerID="94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc" Jul 27 19:56:31.118112 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:56:31.118086 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc\": container with ID starting with 94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc not found: ID does not exist" containerID="94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc" Jul 27 19:56:31.118223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.118127 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc"} err="failed to get container status \"94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc\": rpc error: code = NotFound desc = could not find container \"94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc\": container with ID starting with 94fa573caeb08f6b139f0c5c9bd5142758a097500f2b3cef2821797e0f3d6efc not found: ID does not exist" Jul 27 19:56:31.118223 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.118155 2566 scope.go:117] "RemoveContainer" containerID="f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2" Jul 27 19:56:31.118454 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:56:31.118435 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2\": container with ID starting with f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2 not found: ID does not exist" containerID="f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2" Jul 27 19:56:31.118499 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.118460 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2"} err="failed to get container status \"f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2\": rpc error: code = NotFound desc = could not find container \"f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2\": container with ID starting with f5449ef5d16885295cd6bccd04b5a0337e3b9203eff633905b3fd859234f27a2 not found: ID does not exist" Jul 27 19:56:31.119813 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.119787 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f"] Jul 27 19:56:31.123064 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.123032 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7bbd07-openshift-default-7b9fff55d-dnr4f"] Jul 27 19:56:31.554192 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:31.554147 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" path="/var/lib/kubelet/pods/f1c51423-533f-47c4-981b-37ad3e48aefe/volumes" Jul 27 19:56:48.426994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:48.426943 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg"] Jul 27 19:56:48.428336 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:48.427227 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" containerID="cri-o://e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3" gracePeriod=30 Jul 27 19:56:49.294471 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.294432 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9"] Jul 27 19:56:49.294971 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.294955 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" Jul 27 19:56:49.295033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.294973 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" Jul 27 19:56:49.295033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.294993 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" Jul 27 19:56:49.295033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.294999 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" Jul 27 19:56:49.295033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.295011 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4e5451-4900-4e8c-b813-415bd290adf8" containerName="revoke-keys" Jul 27 19:56:49.295033 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.295018 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4e5451-4900-4e8c-b813-415bd290adf8" containerName="revoke-keys" Jul 27 19:56:49.295227 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.295083 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" Jul 27 19:56:49.295227 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.295091 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1c51423-533f-47c4-981b-37ad3e48aefe" containerName="istio-proxy" Jul 27 19:56:49.295227 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.295100 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4e5451-4900-4e8c-b813-415bd290adf8" containerName="revoke-keys" Jul 27 19:56:49.298377 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.298355 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.300697 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.300671 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-2eb089-openshift-default-dockercfg-pdrvr\"" Jul 27 19:56:49.313348 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.313314 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9"] Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402034 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-envoy\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402109 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-socket\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402155 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-certs\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402241 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-data\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402271 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-credential-socket\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402303 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2qpz\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-kube-api-access-g2qpz\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402345 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-podinfo\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402373 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-token\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.402556 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.402455 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istiod-ca-cert\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.503613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503569 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-data\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.503613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503609 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-credential-socket\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503630 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2qpz\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-kube-api-access-g2qpz\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503752 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-podinfo\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503810 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-token\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503904 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istiod-ca-cert\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.503961 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-envoy\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504017 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-socket\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504115 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504054 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-certs\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504053 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-data\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504302 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-credential-socket\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504362 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-certs\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504468 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-socket\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.504779 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.504738 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istiod-ca-cert\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.506229 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.506207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-envoy\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.506446 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.506428 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-podinfo\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.512742 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.512711 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-token\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.512960 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.512933 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2qpz\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-kube-api-access-g2qpz\") pod \"e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.609189 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.609100 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:49.769237 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.769202 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9"] Jul 27 19:56:49.774976 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.773655 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:56:49.774976 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.773806 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:56:49.774976 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:49.773888 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:56:50.209083 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:50.209048 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" event={"ID":"dfc5df36-1fbb-407c-bc6b-5795de118fcb","Type":"ContainerStarted","Data":"bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8"} Jul 27 19:56:50.209264 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:50.209107 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" event={"ID":"dfc5df36-1fbb-407c-bc6b-5795de118fcb","Type":"ContainerStarted","Data":"74b56f138532366acb6e960d915bc73c193bd49fcfaf0a6720e9b234f45a82e6"} Jul 27 19:56:50.230259 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:50.230193 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podStartSLOduration=1.230169297 podStartE2EDuration="1.230169297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:56:50.227859999 +0000 UTC m=+2383.222356975" watchObservedRunningTime="2026-07-27 19:56:50.230169297 +0000 UTC m=+2383.224666273" Jul 27 19:56:50.610018 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:50.609913 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:50.615546 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:50.615512 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:51.216310 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:51.216273 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:51.217385 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:51.217364 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:56:53.712040 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.712008 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:56:53.746405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746376 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-podinfo\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746411 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-certs\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746452 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-envoy\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746539 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-token\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746564 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54vlm\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-kube-api-access-54vlm\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746644 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746613 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-socket\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746646 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istiod-ca-cert\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746684 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-credential-socket\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746717 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-data\") pod \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\" (UID: \"6ba25fbf-38f8-4d05-9560-621b6a5e94e0\") " Jul 27 19:56:53.746892 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.746733 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:53.747106 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.747061 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:53.747179 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.747109 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.747179 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.747146 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:53.747179 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.747079 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:56:53.747946 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.747911 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-data" (OuterVolumeSpecName: "istio-data") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:53.748947 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.748924 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-token" (OuterVolumeSpecName: "istio-token") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:56:53.749188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.749162 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-kube-api-access-54vlm" (OuterVolumeSpecName: "kube-api-access-54vlm") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "kube-api-access-54vlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:56:53.749263 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.749191 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:56:53.749263 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.749188 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6ba25fbf-38f8-4d05-9560-621b6a5e94e0" (UID: "6ba25fbf-38f8-4d05-9560-621b6a5e94e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:56:53.848264 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848229 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848264 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848257 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848264 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848266 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848264 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848274 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54vlm\" (UniqueName: \"kubernetes.io/projected/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-kube-api-access-54vlm\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848283 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848292 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848309 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:53.848538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:53.848317 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ba25fbf-38f8-4d05-9560-621b6a5e94e0-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:56:54.236451 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.236419 2566 generic.go:356] "Generic (PLEG): container finished" podID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerID="e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3" exitCode=0 Jul 27 19:56:54.236671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.236475 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" Jul 27 19:56:54.236671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.236508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" event={"ID":"6ba25fbf-38f8-4d05-9560-621b6a5e94e0","Type":"ContainerDied","Data":"e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3"} Jul 27 19:56:54.236671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.236563 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" event={"ID":"6ba25fbf-38f8-4d05-9560-621b6a5e94e0","Type":"ContainerDied","Data":"a6582e762069f94c6766ba6d98e03d391b151ceec9574d8235c28139118461a2"} Jul 27 19:56:54.236671 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.236590 2566 scope.go:117] "RemoveContainer" containerID="e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3" Jul 27 19:56:54.250001 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.249982 2566 scope.go:117] "RemoveContainer" containerID="acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0" Jul 27 19:56:54.260284 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.260264 2566 scope.go:117] "RemoveContainer" containerID="e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3" Jul 27 19:56:54.260602 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:56:54.260580 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3\": container with ID starting with e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3 not found: ID does not exist" containerID="e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3" Jul 27 19:56:54.260665 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.260616 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3"} err="failed to get container status \"e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3\": rpc error: code = NotFound desc = could not find container \"e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3\": container with ID starting with e84897535c10f39637655e89964020d6af9b7da948ea32419c8d115a8cabdcf3 not found: ID does not exist" Jul 27 19:56:54.260665 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.260640 2566 scope.go:117] "RemoveContainer" containerID="acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0" Jul 27 19:56:54.260951 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:56:54.260926 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0\": container with ID starting with acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0 not found: ID does not exist" containerID="acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0" Jul 27 19:56:54.261032 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.260954 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0"} err="failed to get container status \"acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0\": rpc error: code = NotFound desc = could not find container \"acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0\": container with ID starting with acb34ff3117b2f3be78187c1a2b10c25fa369872b610ad044d3c2a21312ca9d0 not found: ID does not exist" Jul 27 19:56:54.265938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.265912 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg"] Jul 27 19:56:54.270789 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.270749 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg"] Jul 27 19:56:54.669641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.669527 2566 patch_prober.go:28] interesting pod/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:56:54.669641 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:54.669603 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-4ce65d-openshift-default-855d54ff58-t28dg" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:56:55.548620 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:55.548585 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" path="/var/lib/kubelet/pods/6ba25fbf-38f8-4d05-9560-621b6a5e94e0/volumes" Jul 27 19:56:57.340323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340289 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d"] Jul 27 19:56:57.340772 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340747 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" Jul 27 19:56:57.340825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340787 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" Jul 27 19:56:57.340825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340805 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" Jul 27 19:56:57.340825 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340811 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" Jul 27 19:56:57.340917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340896 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" Jul 27 19:56:57.340917 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.340906 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ba25fbf-38f8-4d05-9560-621b6a5e94e0" containerName="istio-proxy" Jul 27 19:56:57.345504 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.345483 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.348910 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.348887 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-941fec-openshift-default-dockercfg-p8c8d\"" Jul 27 19:56:57.362526 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.362486 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d"] Jul 27 19:56:57.381798 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381749 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e800057b-c67e-4aa9-a687-66c25d935139-istiod-ca-cert\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381807 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-data\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381836 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-envoy\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381860 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e800057b-c67e-4aa9-a687-66c25d935139-istio-podinfo\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381894 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-socket\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381930 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-credential-socket\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382026 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.381954 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-istio-token\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382367 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.382027 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-certs\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.382367 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.382057 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfx4z\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-kube-api-access-rfx4z\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.482898 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.482858 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e800057b-c67e-4aa9-a687-66c25d935139-istiod-ca-cert\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.482898 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.482902 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-data\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483126 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483023 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-envoy\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483126 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483079 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e800057b-c67e-4aa9-a687-66c25d935139-istio-podinfo\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483231 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483125 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-socket\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483231 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483172 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-credential-socket\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483231 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483211 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-istio-token\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483385 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483254 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-certs\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483385 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483280 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-data\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483385 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483293 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfx4z\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-kube-api-access-rfx4z\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483558 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483483 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-socket\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483628 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483554 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-certs\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483683 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483652 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e800057b-c67e-4aa9-a687-66c25d935139-istiod-ca-cert\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.483737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.483696 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-credential-socket\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.485320 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.485299 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-envoy\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.485570 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.485553 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e800057b-c67e-4aa9-a687-66c25d935139-istio-podinfo\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.493137 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.493105 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-istio-token\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.493658 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.493639 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfx4z\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-kube-api-access-rfx4z\") pod \"e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.656452 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.656352 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:57.818408 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.818374 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d"] Jul 27 19:56:57.821214 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:56:57.821185 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode800057b_c67e_4aa9_a687_66c25d935139.slice/crio-0651cd7e1013e9c1f9db2884ac712bd9cbc4b92309f5d4747467721a932e9425 WatchSource:0}: Error finding container 0651cd7e1013e9c1f9db2884ac712bd9cbc4b92309f5d4747467721a932e9425: Status 404 returned error can't find the container with id 0651cd7e1013e9c1f9db2884ac712bd9cbc4b92309f5d4747467721a932e9425 Jul 27 19:56:57.823487 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.823445 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:56:57.823576 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.823537 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:56:57.823613 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:57.823581 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:56:58.267286 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:58.267245 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" event={"ID":"e800057b-c67e-4aa9-a687-66c25d935139","Type":"ContainerStarted","Data":"c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd"} Jul 27 19:56:58.267509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:58.267299 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" event={"ID":"e800057b-c67e-4aa9-a687-66c25d935139","Type":"ContainerStarted","Data":"0651cd7e1013e9c1f9db2884ac712bd9cbc4b92309f5d4747467721a932e9425"} Jul 27 19:56:58.350433 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:58.350376 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podStartSLOduration=1.350355057 podStartE2EDuration="1.350355057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:56:58.348558794 +0000 UTC m=+2391.343055800" watchObservedRunningTime="2026-07-27 19:56:58.350355057 +0000 UTC m=+2391.344852031" Jul 27 19:56:58.657580 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:58.657481 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:58.663011 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:58.662981 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:59.273468 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:59.273436 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:56:59.274538 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:56:59.274517 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:57:01.352731 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.352695 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485"] Jul 27 19:57:01.356718 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.356698 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.360560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.360537 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-941fec-dockercfg-fmdmh\"" Jul 27 19:57:01.360898 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.360884 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-941fec\"" Jul 27 19:57:01.375146 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.375120 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485"] Jul 27 19:57:01.428792 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.428746 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkvnp\" (UniqueName: \"kubernetes.io/projected/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-kube-api-access-dkvnp\") pod \"payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.428997 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.428974 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.530736 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.530677 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.530966 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.530840 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkvnp\" (UniqueName: \"kubernetes.io/projected/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-kube-api-access-dkvnp\") pod \"payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.531818 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.531783 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.539775 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.539729 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkvnp\" (UniqueName: \"kubernetes.io/projected/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-kube-api-access-dkvnp\") pod \"payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.668007 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.667914 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:01.873642 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:01.873607 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485"] Jul 27 19:57:01.875936 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:57:01.875898 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed1a98f_71e6_49f0_a5ac_d3e9b9ff46b2.slice/crio-ca6534aee2d2580228ad7f32d6072b1b5b97ed75f93edd77d4de692a151542f1 WatchSource:0}: Error finding container ca6534aee2d2580228ad7f32d6072b1b5b97ed75f93edd77d4de692a151542f1: Status 404 returned error can't find the container with id ca6534aee2d2580228ad7f32d6072b1b5b97ed75f93edd77d4de692a151542f1 Jul 27 19:57:02.293734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:02.293639 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" event={"ID":"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2","Type":"ContainerStarted","Data":"728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7"} Jul 27 19:57:02.293734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:02.293707 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:02.293734 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:02.293732 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" event={"ID":"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2","Type":"ContainerStarted","Data":"ca6534aee2d2580228ad7f32d6072b1b5b97ed75f93edd77d4de692a151542f1"} Jul 27 19:57:02.316778 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:02.316693 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" podStartSLOduration=1.316668463 podStartE2EDuration="1.316668463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:57:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:57:02.313630397 +0000 UTC m=+2395.308127375" watchObservedRunningTime="2026-07-27 19:57:02.316668463 +0000 UTC m=+2395.311165440" Jul 27 19:57:07.258996 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:07.258944 2566 patch_prober.go:28] interesting pod/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 27 19:57:07.259509 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:07.259033 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 27 19:57:07.752108 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:07.752082 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:57:07.754688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:07.754666 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 19:57:09.344994 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:09.344957 2566 generic.go:356] "Generic (PLEG): container finished" podID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerID="bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8" exitCode=0 Jul 27 19:57:09.345455 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:09.345027 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" event={"ID":"dfc5df36-1fbb-407c-bc6b-5795de118fcb","Type":"ContainerDied","Data":"bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8"} Jul 27 19:57:09.345502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:09.345471 2566 scope.go:117] "RemoveContainer" containerID="bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8" Jul 27 19:57:09.346536 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:09.346508 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:57:09.346600 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:09.346579 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:57:09.346636 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:09.346610 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:57:10.352954 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:10.352919 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" event={"ID":"dfc5df36-1fbb-407c-bc6b-5795de118fcb","Type":"ContainerStarted","Data":"0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8"} Jul 27 19:57:10.609469 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:10.609367 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:57:10.609668 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:10.609638 2566 patch_prober.go:28] interesting pod/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 27 19:57:10.609749 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:10.609701 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 27 19:57:11.609698 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:11.609662 2566 patch_prober.go:28] interesting pod/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 27 19:57:11.610249 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:11.609718 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 27 19:57:12.610225 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:12.610188 2566 patch_prober.go:28] interesting pod/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 27 19:57:12.610855 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:12.610824 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 27 19:57:13.302322 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:13.302291 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:13.610404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:13.610293 2566 patch_prober.go:28] interesting pod/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Jul 27 19:57:13.610404 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:13.610362 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Jul 27 19:57:14.620340 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:14.620298 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:57:14.622668 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:14.622644 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:57:14.628612 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:14.628583 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:57:19.419216 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:19.419177 2566 generic.go:356] "Generic (PLEG): container finished" podID="e800057b-c67e-4aa9-a687-66c25d935139" containerID="c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd" exitCode=0 Jul 27 19:57:19.419738 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:19.419273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" event={"ID":"e800057b-c67e-4aa9-a687-66c25d935139","Type":"ContainerDied","Data":"c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd"} Jul 27 19:57:19.419878 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:19.419860 2566 scope.go:117] "RemoveContainer" containerID="c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd" Jul 27 19:57:19.421065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:19.421033 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:57:19.421153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:19.421099 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:57:19.421153 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:19.421132 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:57:20.443877 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:20.443837 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" event={"ID":"e800057b-c67e-4aa9-a687-66c25d935139","Type":"ContainerStarted","Data":"34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176"} Jul 27 19:57:20.656907 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:20.656876 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:57:20.657077 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:20.657055 2566 patch_prober.go:28] interesting pod/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 27 19:57:20.657123 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:20.657098 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 27 19:57:21.657104 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:21.657061 2566 patch_prober.go:28] interesting pod/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 27 19:57:21.657597 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:21.657137 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 27 19:57:22.657853 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:22.657815 2566 patch_prober.go:28] interesting pod/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 27 19:57:22.658267 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:22.657880 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 27 19:57:23.657688 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:23.657643 2566 patch_prober.go:28] interesting pod/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Jul 27 19:57:23.657920 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:23.657718 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Jul 27 19:57:24.661086 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:24.661055 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:57:24.661563 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:24.661541 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:57:24.662142 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:24.662124 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:57:39.633611 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:39.633520 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9"] Jul 27 19:57:39.634120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:39.633854 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" containerID="cri-o://0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8" gracePeriod=30 Jul 27 19:57:44.624668 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:44.624625 2566 patch_prober.go:28] interesting pod/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:57:44.625206 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:44.624691 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:57:44.908249 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:44.908219 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:57:45.002625 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002588 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-podinfo\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.002852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002637 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-socket\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.002852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002671 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-credential-socket\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.002852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002703 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istiod-ca-cert\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.002852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002728 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-certs\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.002852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002795 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-envoy\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.003152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002861 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2qpz\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-kube-api-access-g2qpz\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.003152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002933 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-data\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.003152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.002995 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-token\") pod \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\" (UID: \"dfc5df36-1fbb-407c-bc6b-5795de118fcb\") " Jul 27 19:57:45.003152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003063 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:57:45.003152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003080 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:57:45.003152 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003090 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:57:45.003553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003173 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:57:45.003861 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003839 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.003938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003867 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.003938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003883 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.003938 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.003896 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.004072 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.004018 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-data" (OuterVolumeSpecName: "istio-data") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:57:45.005065 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.005034 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:57:45.005186 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.005093 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:57:45.005506 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.005482 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-kube-api-access-g2qpz" (OuterVolumeSpecName: "kube-api-access-g2qpz") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "kube-api-access-g2qpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:57:45.005612 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.005513 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-token" (OuterVolumeSpecName: "istio-token") pod "dfc5df36-1fbb-407c-bc6b-5795de118fcb" (UID: "dfc5df36-1fbb-407c-bc6b-5795de118fcb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:57:45.105150 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.105112 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.105150 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.105145 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2qpz\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-kube-api-access-g2qpz\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.105150 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.105155 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.105396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.105165 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.105396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.105172 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfc5df36-1fbb-407c-bc6b-5795de118fcb-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:45.592628 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.592591 2566 generic.go:356] "Generic (PLEG): container finished" podID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerID="0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8" exitCode=0 Jul 27 19:57:45.592835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.592639 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" Jul 27 19:57:45.592835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.592675 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" event={"ID":"dfc5df36-1fbb-407c-bc6b-5795de118fcb","Type":"ContainerDied","Data":"0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8"} Jul 27 19:57:45.592835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.592730 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9" event={"ID":"dfc5df36-1fbb-407c-bc6b-5795de118fcb","Type":"ContainerDied","Data":"74b56f138532366acb6e960d915bc73c193bd49fcfaf0a6720e9b234f45a82e6"} Jul 27 19:57:45.592835 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.592756 2566 scope.go:117] "RemoveContainer" containerID="0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8" Jul 27 19:57:45.602240 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.602220 2566 scope.go:117] "RemoveContainer" containerID="bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8" Jul 27 19:57:45.612072 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.612053 2566 scope.go:117] "RemoveContainer" containerID="0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8" Jul 27 19:57:45.612360 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:57:45.612340 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8\": container with ID starting with 0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8 not found: ID does not exist" containerID="0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8" Jul 27 19:57:45.612403 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.612369 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8"} err="failed to get container status \"0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8\": rpc error: code = NotFound desc = could not find container \"0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8\": container with ID starting with 0177363d06bd4c54d29a8c18df9bf983d92d215cf40334191a115392bd63a3b8 not found: ID does not exist" Jul 27 19:57:45.612403 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.612386 2566 scope.go:117] "RemoveContainer" containerID="bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8" Jul 27 19:57:45.612668 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:57:45.612646 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8\": container with ID starting with bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8 not found: ID does not exist" containerID="bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8" Jul 27 19:57:45.612718 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.612676 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8"} err="failed to get container status \"bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8\": rpc error: code = NotFound desc = could not find container \"bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8\": container with ID starting with bdc2153f127a26d9f5d7c5505441c2fe31c6e224a0fe6332bd104b58e13adfc8 not found: ID does not exist" Jul 27 19:57:45.636591 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.636551 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9"] Jul 27 19:57:45.644957 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:45.644925 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-2eb089-openshift-default-778769f6d8-lqxp9"] Jul 27 19:57:47.547548 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:47.547512 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" path="/var/lib/kubelet/pods/dfc5df36-1fbb-407c-bc6b-5795de118fcb/volumes" Jul 27 19:57:55.629648 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:55.629605 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485"] Jul 27 19:57:55.630708 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:55.629888 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" podUID="9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" containerName="payload-pre-processing" containerID="cri-o://728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7" gracePeriod=30 Jul 27 19:57:55.909779 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:55.909728 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:56.021551 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.021507 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-plugins-config-volume\") pod \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " Jul 27 19:57:56.021729 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.021686 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dkvnp\" (UniqueName: \"kubernetes.io/projected/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-kube-api-access-dkvnp\") pod \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\" (UID: \"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2\") " Jul 27 19:57:56.022020 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.021990 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" (UID: "9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:57:56.023833 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.023807 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-kube-api-access-dkvnp" (OuterVolumeSpecName: "kube-api-access-dkvnp") pod "9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" (UID: "9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2"). InnerVolumeSpecName "kube-api-access-dkvnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:57:56.122982 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.122947 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:56.122982 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.122976 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkvnp\" (UniqueName: \"kubernetes.io/projected/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2-kube-api-access-dkvnp\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:57:56.660978 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.660937 2566 generic.go:356] "Generic (PLEG): container finished" podID="9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" containerID="728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7" exitCode=0 Jul 27 19:57:56.661392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.660987 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" Jul 27 19:57:56.661392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.661020 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" event={"ID":"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2","Type":"ContainerDied","Data":"728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7"} Jul 27 19:57:56.661392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.661078 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485" event={"ID":"9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2","Type":"ContainerDied","Data":"ca6534aee2d2580228ad7f32d6072b1b5b97ed75f93edd77d4de692a151542f1"} Jul 27 19:57:56.661392 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.661103 2566 scope.go:117] "RemoveContainer" containerID="728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7" Jul 27 19:57:56.671029 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.671007 2566 scope.go:117] "RemoveContainer" containerID="728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7" Jul 27 19:57:56.671387 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:57:56.671355 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7\": container with ID starting with 728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7 not found: ID does not exist" containerID="728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7" Jul 27 19:57:56.671440 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.671399 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7"} err="failed to get container status \"728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7\": rpc error: code = NotFound desc = could not find container \"728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7\": container with ID starting with 728804792f847f89af19dbf74cba4b870b4e3cd5b993f02c85537656426ecab7 not found: ID does not exist" Jul 27 19:57:56.685774 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.685723 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485"] Jul 27 19:57:56.688270 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:56.688237 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-941fec-58fc7d8db4-qs485"] Jul 27 19:57:57.549364 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:57:57.549325 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" path="/var/lib/kubelet/pods/9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2/volumes" Jul 27 19:58:06.769967 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:06.769930 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d"] Jul 27 19:58:06.770431 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:06.770155 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" containerID="cri-o://34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176" gracePeriod=30 Jul 27 19:58:07.619608 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.619571 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg"] Jul 27 19:58:07.620316 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620290 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" Jul 27 19:58:07.620316 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620316 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" Jul 27 19:58:07.620537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620348 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" containerName="payload-pre-processing" Jul 27 19:58:07.620537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620357 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" containerName="payload-pre-processing" Jul 27 19:58:07.620537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620488 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed1a98f-71e6-49f0-a5ac-d3e9b9ff46b2" containerName="payload-pre-processing" Jul 27 19:58:07.620537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620502 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" Jul 27 19:58:07.620537 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.620513 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" Jul 27 19:58:07.625701 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.625674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.628119 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.628091 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-4289a9-openshift-default-dockercfg-8gk4s\"" Jul 27 19:58:07.637128 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.637095 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg"] Jul 27 19:58:07.739553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739512 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-certs\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.739553 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739555 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-podinfo\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.739831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739584 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-credential-socket\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.739831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739658 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-socket\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.739831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739787 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-data\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.739831 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739826 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istiod-ca-cert\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.740045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739919 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-envoy\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.740045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.739964 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x2rs\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-kube-api-access-6x2rs\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.740045 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.740002 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-token\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841257 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841226 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x2rs\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-kube-api-access-6x2rs\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841267 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-token\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841303 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-certs\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841327 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-podinfo\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841365 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-credential-socket\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841399 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-socket\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841584 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-data\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841615 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istiod-ca-cert\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.841713 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841694 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-envoy\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.842188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841747 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-certs\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.842188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841821 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-credential-socket\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.842188 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.841863 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-socket\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.842419 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.842386 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istiod-ca-cert\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.842615 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.842593 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-data\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.843777 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.843737 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-envoy\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.843863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.843809 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-podinfo\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.854504 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.854389 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-token\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.854504 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.854451 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x2rs\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-kube-api-access-6x2rs\") pod \"e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:07.915614 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.915521 2566 scope.go:117] "RemoveContainer" containerID="70dab5d6746596fb9fdb7675cca4ef4c67afab6d58fad6dedf87027c52044694" Jul 27 19:58:07.939161 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:07.939089 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:08.097341 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.097300 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg"] Jul 27 19:58:08.100530 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:58:08.100502 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7c42ae_fa6f_41d7_8e49_c1ea86ba1696.slice/crio-031e47aa6d13e67eb6cff9fd8844b6ae15d4c6c3dd11778850214773eb79f4a5 WatchSource:0}: Error finding container 031e47aa6d13e67eb6cff9fd8844b6ae15d4c6c3dd11778850214773eb79f4a5: Status 404 returned error can't find the container with id 031e47aa6d13e67eb6cff9fd8844b6ae15d4c6c3dd11778850214773eb79f4a5 Jul 27 19:58:08.103057 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.103018 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:08.103133 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.103096 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:08.103169 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.103130 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:08.743784 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.743728 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" event={"ID":"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696","Type":"ContainerStarted","Data":"8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce"} Jul 27 19:58:08.744000 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.743815 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" event={"ID":"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696","Type":"ContainerStarted","Data":"031e47aa6d13e67eb6cff9fd8844b6ae15d4c6c3dd11778850214773eb79f4a5"} Jul 27 19:58:08.768586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.768532 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podStartSLOduration=1.768516319 podStartE2EDuration="1.768516319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:08.767868425 +0000 UTC m=+2461.762365411" watchObservedRunningTime="2026-07-27 19:58:08.768516319 +0000 UTC m=+2461.763013294" Jul 27 19:58:08.939703 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:08.939668 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:09.664242 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:09.664201 2566 patch_prober.go:28] interesting pod/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 27 19:58:09.664416 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:09.664276 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 27 19:58:09.945052 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:09.945021 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:10.756380 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:10.756343 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:10.757294 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:10.757273 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:12.065445 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.065418 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:58:12.190335 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190296 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-socket\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190348 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-credential-socket\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190373 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-istio-token\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190396 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfx4z\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-kube-api-access-rfx4z\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190442 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-envoy\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190461 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-data\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190518 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190519 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e800057b-c67e-4aa9-a687-66c25d935139-istio-podinfo\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190556 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-certs\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190590 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e800057b-c67e-4aa9-a687-66c25d935139-istiod-ca-cert\") pod \"e800057b-c67e-4aa9-a687-66c25d935139\" (UID: \"e800057b-c67e-4aa9-a687-66c25d935139\") " Jul 27 19:58:12.190863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190631 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:58:12.190863 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190659 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:58:12.191084 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.190921 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:58:12.191084 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.191020 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e800057b-c67e-4aa9-a687-66c25d935139-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 19:58:12.191187 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.191063 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.191187 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.191140 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.191187 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.191157 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.191731 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.191705 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-data" (OuterVolumeSpecName: "istio-data") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:58:12.192893 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.192870 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e800057b-c67e-4aa9-a687-66c25d935139-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 19:58:12.192991 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.192949 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-kube-api-access-rfx4z" (OuterVolumeSpecName: "kube-api-access-rfx4z") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "kube-api-access-rfx4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:58:12.193071 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.193057 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 19:58:12.193400 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.193387 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-istio-token" (OuterVolumeSpecName: "istio-token") pod "e800057b-c67e-4aa9-a687-66c25d935139" (UID: "e800057b-c67e-4aa9-a687-66c25d935139"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 19:58:12.300382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.294011 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e800057b-c67e-4aa9-a687-66c25d935139-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.300382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.294056 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.300382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.294071 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfx4z\" (UniqueName: \"kubernetes.io/projected/e800057b-c67e-4aa9-a687-66c25d935139-kube-api-access-rfx4z\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.300382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.294088 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.300382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.294101 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e800057b-c67e-4aa9-a687-66c25d935139-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.300382 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.294121 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e800057b-c67e-4aa9-a687-66c25d935139-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 19:58:12.768530 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.768489 2566 generic.go:356] "Generic (PLEG): container finished" podID="e800057b-c67e-4aa9-a687-66c25d935139" containerID="34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176" exitCode=0 Jul 27 19:58:12.768754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.768543 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" event={"ID":"e800057b-c67e-4aa9-a687-66c25d935139","Type":"ContainerDied","Data":"34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176"} Jul 27 19:58:12.768754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.768640 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" event={"ID":"e800057b-c67e-4aa9-a687-66c25d935139","Type":"ContainerDied","Data":"0651cd7e1013e9c1f9db2884ac712bd9cbc4b92309f5d4747467721a932e9425"} Jul 27 19:58:12.768754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.768662 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d" Jul 27 19:58:12.768754 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.768685 2566 scope.go:117] "RemoveContainer" containerID="34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176" Jul 27 19:58:12.779114 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.779094 2566 scope.go:117] "RemoveContainer" containerID="c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd" Jul 27 19:58:12.788782 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.788744 2566 scope.go:117] "RemoveContainer" containerID="34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176" Jul 27 19:58:12.789091 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:58:12.789070 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176\": container with ID starting with 34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176 not found: ID does not exist" containerID="34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176" Jul 27 19:58:12.789136 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.789102 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176"} err="failed to get container status \"34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176\": rpc error: code = NotFound desc = could not find container \"34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176\": container with ID starting with 34a6cc813c3622f6410bdda7ac4fad79133955b5984f23c4c170e53fd9b99176 not found: ID does not exist" Jul 27 19:58:12.789136 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.789120 2566 scope.go:117] "RemoveContainer" containerID="c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd" Jul 27 19:58:12.789368 ip-10-0-132-244 kubenswrapper[2566]: E0727 19:58:12.789354 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd\": container with ID starting with c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd not found: ID does not exist" containerID="c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd" Jul 27 19:58:12.789411 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.789371 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd"} err="failed to get container status \"c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd\": rpc error: code = NotFound desc = could not find container \"c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd\": container with ID starting with c4eac7ebe38e02786739238476d92fe6f1e8112e69d49fe17c82138e5ceb76dd not found: ID does not exist" Jul 27 19:58:12.796221 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.796192 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d"] Jul 27 19:58:12.804080 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:12.804054 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-941fec-openshift-default-b7b88ffd6-2dq5d"] Jul 27 19:58:13.548949 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:13.548914 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e800057b-c67e-4aa9-a687-66c25d935139" path="/var/lib/kubelet/pods/e800057b-c67e-4aa9-a687-66c25d935139/volumes" Jul 27 19:58:15.803850 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.803816 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6"] Jul 27 19:58:15.804310 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804294 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" Jul 27 19:58:15.804354 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804311 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfc5df36-1fbb-407c-bc6b-5795de118fcb" containerName="istio-proxy" Jul 27 19:58:15.804354 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804336 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" Jul 27 19:58:15.804354 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804341 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" Jul 27 19:58:15.804354 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804350 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" Jul 27 19:58:15.804482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804357 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" Jul 27 19:58:15.804482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804427 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" Jul 27 19:58:15.804482 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.804435 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e800057b-c67e-4aa9-a687-66c25d935139" containerName="istio-proxy" Jul 27 19:58:15.807697 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.807677 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.811122 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.811093 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-f99bfd-openshift-default-dockercfg-587z6\"" Jul 27 19:58:15.834465 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.834429 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6"] Jul 27 19:58:15.930291 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930239 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-credential-socket\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930478 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930350 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-envoy\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930478 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930386 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e646924b-5519-419a-96b9-07f77f0a132c-istio-podinfo\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930478 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930432 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-istio-token\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930475 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-certs\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930495 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfltq\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-kube-api-access-gfltq\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930518 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e646924b-5519-419a-96b9-07f77f0a132c-istiod-ca-cert\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930586 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930568 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-data\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:15.930715 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:15.930599 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-socket\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031640 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031606 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e646924b-5519-419a-96b9-07f77f0a132c-istio-podinfo\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031640 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031652 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-istio-token\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031686 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-certs\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfltq\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-kube-api-access-gfltq\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031725 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e646924b-5519-419a-96b9-07f77f0a132c-istiod-ca-cert\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031790 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-data\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031832 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-socket\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.031918 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031866 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-credential-socket\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.032121 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.031954 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-envoy\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.032239 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.032207 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-certs\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.032444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.032345 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-socket\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.032444 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.032429 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-data\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.032569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.032457 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-credential-socket\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.032569 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.032509 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e646924b-5519-419a-96b9-07f77f0a132c-istiod-ca-cert\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.034213 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.034182 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-envoy\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.034396 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.034379 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e646924b-5519-419a-96b9-07f77f0a132c-istio-podinfo\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.053244 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.053216 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfltq\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-kube-api-access-gfltq\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.058545 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.058480 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-istio-token\") pod \"e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.121689 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.121640 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:16.289132 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.289097 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6"] Jul 27 19:58:16.293860 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:58:16.293829 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode646924b_5519_419a_96b9_07f77f0a132c.slice/crio-441e85999cf9572012a111c53b777a0102468cb3cf1468e280525b04d5a46ba7 WatchSource:0}: Error finding container 441e85999cf9572012a111c53b777a0102468cb3cf1468e280525b04d5a46ba7: Status 404 returned error can't find the container with id 441e85999cf9572012a111c53b777a0102468cb3cf1468e280525b04d5a46ba7 Jul 27 19:58:16.296123 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.296090 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:16.296192 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.296160 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:16.296192 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.296188 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:16.795008 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.794971 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" event={"ID":"e646924b-5519-419a-96b9-07f77f0a132c","Type":"ContainerStarted","Data":"44307ceb87fc7b371221c162c0af4b6aa4baf6f166ec9fe527037046fd6df6d7"} Jul 27 19:58:16.795233 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.795063 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" event={"ID":"e646924b-5519-419a-96b9-07f77f0a132c","Type":"ContainerStarted","Data":"441e85999cf9572012a111c53b777a0102468cb3cf1468e280525b04d5a46ba7"} Jul 27 19:58:16.833280 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:16.833221 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podStartSLOduration=1.83320309 podStartE2EDuration="1.83320309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:16.831948383 +0000 UTC m=+2469.826445359" watchObservedRunningTime="2026-07-27 19:58:16.83320309 +0000 UTC m=+2469.827700136" Jul 27 19:58:17.123029 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:17.122931 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:17.135279 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:17.135243 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:17.800737 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:17.800704 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:17.801729 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:17.801710 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:19.868400 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.868360 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r"] Jul 27 19:58:19.872488 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.872469 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:19.874973 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.874951 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-f99bfd-dockercfg-kfzjw\"" Jul 27 19:58:19.875074 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.874957 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-f99bfd\"" Jul 27 19:58:19.885247 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.885216 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r"] Jul 27 19:58:19.977949 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.977909 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxn79\" (UniqueName: \"kubernetes.io/projected/11cabc00-a25e-4bbc-886f-34721cdc8f91-kube-api-access-bxn79\") pod \"payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:19.978156 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:19.978064 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cabc00-a25e-4bbc-886f-34721cdc8f91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.079452 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.079417 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cabc00-a25e-4bbc-886f-34721cdc8f91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.079637 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.079500 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxn79\" (UniqueName: \"kubernetes.io/projected/11cabc00-a25e-4bbc-886f-34721cdc8f91-kube-api-access-bxn79\") pod \"payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.080384 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.080357 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cabc00-a25e-4bbc-886f-34721cdc8f91-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.095956 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.095924 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxn79\" (UniqueName: \"kubernetes.io/projected/11cabc00-a25e-4bbc-886f-34721cdc8f91-kube-api-access-bxn79\") pod \"payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.185692 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.185652 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.381096 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.380884 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r"] Jul 27 19:58:20.383615 ip-10-0-132-244 kubenswrapper[2566]: W0727 19:58:20.383582 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cabc00_a25e_4bbc_886f_34721cdc8f91.slice/crio-372261dd0e01ea36c343cf3cccc93751ddbcc1cdac94f8674c8c3f5d2e0cfc78 WatchSource:0}: Error finding container 372261dd0e01ea36c343cf3cccc93751ddbcc1cdac94f8674c8c3f5d2e0cfc78: Status 404 returned error can't find the container with id 372261dd0e01ea36c343cf3cccc93751ddbcc1cdac94f8674c8c3f5d2e0cfc78 Jul 27 19:58:20.835120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.835010 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" event={"ID":"11cabc00-a25e-4bbc-886f-34721cdc8f91","Type":"ContainerStarted","Data":"63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306"} Jul 27 19:58:20.835120 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.835088 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" event={"ID":"11cabc00-a25e-4bbc-886f-34721cdc8f91","Type":"ContainerStarted","Data":"372261dd0e01ea36c343cf3cccc93751ddbcc1cdac94f8674c8c3f5d2e0cfc78"} Jul 27 19:58:20.835992 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.835965 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:20.857425 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:20.857369 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" podStartSLOduration=1.857350715 podStartE2EDuration="1.857350715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 19:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 19:58:20.854470126 +0000 UTC m=+2473.848967102" watchObservedRunningTime="2026-07-27 19:58:20.857350715 +0000 UTC m=+2473.851847689" Jul 27 19:58:24.862543 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:24.862509 2566 generic.go:356] "Generic (PLEG): container finished" podID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerID="8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce" exitCode=0 Jul 27 19:58:24.862999 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:24.862544 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" event={"ID":"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696","Type":"ContainerDied","Data":"8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce"} Jul 27 19:58:24.863053 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:24.863038 2566 scope.go:117] "RemoveContainer" containerID="8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce" Jul 27 19:58:24.864282 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:24.864252 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:24.864379 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:24.864335 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:24.864442 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:24.864386 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:25.756491 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:25.756456 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:25.869997 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:25.869961 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" event={"ID":"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696","Type":"ContainerStarted","Data":"259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669"} Jul 27 19:58:25.940247 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:25.940210 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:25.941405 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:25.941376 2566 patch_prober.go:28] interesting pod/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 27 19:58:25.941560 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:25.941434 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 27 19:58:26.939838 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:26.939799 2566 patch_prober.go:28] interesting pod/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 27 19:58:26.940564 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:26.940534 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 27 19:58:27.939626 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:27.939581 2566 patch_prober.go:28] interesting pod/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 27 19:58:27.939870 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:27.939647 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 27 19:58:28.939861 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:28.939814 2566 patch_prober.go:28] interesting pod/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 27 19:58:28.940369 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:28.939930 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 27 19:58:29.940356 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:29.939984 2566 patch_prober.go:28] interesting pod/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Jul 27 19:58:29.940356 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:29.940037 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Jul 27 19:58:30.944660 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:30.944629 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:30.945183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:30.945034 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:30.945853 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:30.945825 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 19:58:32.801964 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.801932 2566 patch_prober.go:28] interesting pod/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 27 19:58:32.802412 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.801987 2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 27 19:58:32.851820 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.851791 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 19:58:32.915297 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.915260 2566 generic.go:356] "Generic (PLEG): container finished" podID="e646924b-5519-419a-96b9-07f77f0a132c" containerID="44307ceb87fc7b371221c162c0af4b6aa4baf6f166ec9fe527037046fd6df6d7" exitCode=0 Jul 27 19:58:32.915502 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.915329 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" event={"ID":"e646924b-5519-419a-96b9-07f77f0a132c","Type":"ContainerDied","Data":"44307ceb87fc7b371221c162c0af4b6aa4baf6f166ec9fe527037046fd6df6d7"} Jul 27 19:58:32.916210 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.916191 2566 scope.go:117] "RemoveContainer" containerID="44307ceb87fc7b371221c162c0af4b6aa4baf6f166ec9fe527037046fd6df6d7" Jul 27 19:58:32.917665 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.917631 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:32.917805 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.917720 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:32.917805 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:32.917755 2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 27 19:58:33.931512 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:33.931459 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" event={"ID":"e646924b-5519-419a-96b9-07f77f0a132c","Type":"ContainerStarted","Data":"5af8e854b00949fd154a7c139b6a4f7ff091491847547952d2f590f475509c93"} Jul 27 19:58:34.122852 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:34.122817 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:34.123323 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:34.123293 2566 patch_prober.go:28] interesting pod/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 27 19:58:34.123463 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:34.123355 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 27 19:58:35.122214 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:35.122157 2566 patch_prober.go:28] interesting pod/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 27 19:58:35.122908 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:35.122874 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 27 19:58:36.123155 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:36.123117 2566 patch_prober.go:28] interesting pod/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 27 19:58:36.123804 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:36.123748 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 27 19:58:37.122864 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:37.122824 2566 patch_prober.go:28] interesting pod/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Jul 27 19:58:37.123183 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:37.123152 2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Jul 27 19:58:38.126258 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:38.126227 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:38.126649 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:38.126493 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 19:58:38.127285 ip-10-0-132-244 kubenswrapper[2566]: I0727 19:58:38.127265 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 20:00:00.154793 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.154740 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn"] Jul 27 20:00:00.158431 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.158409 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n"] Jul 27 20:00:00.158542 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.158520 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" Jul 27 20:00:00.160738 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.160713 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 20:00:00.162243 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.162223 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" Jul 27 20:00:00.168815 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.168793 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn"] Jul 27 20:00:00.171728 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.171701 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n"] Jul 27 20:00:00.172016 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.171994 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" Jul 27 20:00:00.179792 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.179747 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" Jul 27 20:00:00.394583 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:00:00.394549 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2310426_2c7f_4d4e_8ba2_4a8be36174be.slice/crio-3876df4caa0e26e8c79b713d53fa1617a665434dea8847f14d012b6d0a93a21a WatchSource:0}: Error finding container 3876df4caa0e26e8c79b713d53fa1617a665434dea8847f14d012b6d0a93a21a: Status 404 returned error can't find the container with id 3876df4caa0e26e8c79b713d53fa1617a665434dea8847f14d012b6d0a93a21a Jul 27 20:00:00.394749 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.394691 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn"] Jul 27 20:00:00.450123 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.450089 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" event={"ID":"f2310426-2c7f-4d4e-8ba2-4a8be36174be","Type":"ContainerStarted","Data":"3876df4caa0e26e8c79b713d53fa1617a665434dea8847f14d012b6d0a93a21a"} Jul 27 20:00:00.567885 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:00.567860 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n"] Jul 27 20:00:00.570103 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:00:00.570075 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71c5dbe_30d4_427e_a841_b3672c659203.slice/crio-176ec8c0dd402c83102c0f08e65ddc65aa9cc0b6b24de15cc0fce6f654ed34e4 WatchSource:0}: Error finding container 176ec8c0dd402c83102c0f08e65ddc65aa9cc0b6b24de15cc0fce6f654ed34e4: Status 404 returned error can't find the container with id 176ec8c0dd402c83102c0f08e65ddc65aa9cc0b6b24de15cc0fce6f654ed34e4 Jul 27 20:00:01.458298 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:01.458260 2566 generic.go:356] "Generic (PLEG): container finished" podID="f2310426-2c7f-4d4e-8ba2-4a8be36174be" containerID="2aee505923992c3afc9cf427d32d797d30dab3a34fd5c7ea82908cd8bd2aa9e0" exitCode=0 Jul 27 20:00:01.459169 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:01.458301 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" event={"ID":"f2310426-2c7f-4d4e-8ba2-4a8be36174be","Type":"ContainerDied","Data":"2aee505923992c3afc9cf427d32d797d30dab3a34fd5c7ea82908cd8bd2aa9e0"} Jul 27 20:00:01.460084 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:01.460051 2566 generic.go:356] "Generic (PLEG): container finished" podID="f71c5dbe-30d4-427e-a841-b3672c659203" containerID="2e857fa939e20cf526efc7012bb014f6141fe857e891e6c86dba92a7035ca33d" exitCode=0 Jul 27 20:00:01.460235 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:01.460113 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" event={"ID":"f71c5dbe-30d4-427e-a841-b3672c659203","Type":"ContainerDied","Data":"2e857fa939e20cf526efc7012bb014f6141fe857e891e6c86dba92a7035ca33d"} Jul 27 20:00:01.460235 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:01.460151 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" event={"ID":"f71c5dbe-30d4-427e-a841-b3672c659203","Type":"ContainerStarted","Data":"176ec8c0dd402c83102c0f08e65ddc65aa9cc0b6b24de15cc0fce6f654ed34e4"} Jul 27 20:00:02.615481 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:02.615453 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" Jul 27 20:00:02.641232 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:02.641206 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" Jul 27 20:00:03.474244 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:03.474212 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" Jul 27 20:00:03.474447 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:03.474242 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n" event={"ID":"f71c5dbe-30d4-427e-a841-b3672c659203","Type":"ContainerDied","Data":"176ec8c0dd402c83102c0f08e65ddc65aa9cc0b6b24de15cc0fce6f654ed34e4"} Jul 27 20:00:03.474447 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:03.474302 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="176ec8c0dd402c83102c0f08e65ddc65aa9cc0b6b24de15cc0fce6f654ed34e4" Jul 27 20:00:03.475705 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:03.475687 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" Jul 27 20:00:03.475705 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:03.475698 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29753040-87ndn" event={"ID":"f2310426-2c7f-4d4e-8ba2-4a8be36174be","Type":"ContainerDied","Data":"3876df4caa0e26e8c79b713d53fa1617a665434dea8847f14d012b6d0a93a21a"} Jul 27 20:00:03.475871 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:00:03.475738 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3876df4caa0e26e8c79b713d53fa1617a665434dea8847f14d012b6d0a93a21a" Jul 27 20:02:07.790324 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:02:07.790290 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:02:07.796435 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:02:07.796409 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:02:07.988724 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:02:07.988690 2566 scope.go:117] "RemoveContainer" containerID="b5f2e594254e7ded40408cde0ce155229c014fc450b033c7afec54db6ccfc64c" Jul 27 20:07:07.830384 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:07.830354 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:07:07.836905 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:07.836878 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:07:44.625554 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.625508 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-140224303351344"] Jul 27 20:07:44.626108 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.626028 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71c5dbe-30d4-427e-a841-b3672c659203" containerName="cleanup" Jul 27 20:07:44.626108 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.626041 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71c5dbe-30d4-427e-a841-b3672c659203" containerName="cleanup" Jul 27 20:07:44.626108 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.626054 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2310426-2c7f-4d4e-8ba2-4a8be36174be" containerName="cleanup" Jul 27 20:07:44.626108 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.626060 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2310426-2c7f-4d4e-8ba2-4a8be36174be" containerName="cleanup" Jul 27 20:07:44.626246 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.626159 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2310426-2c7f-4d4e-8ba2-4a8be36174be" containerName="cleanup" Jul 27 20:07:44.626246 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.626168 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71c5dbe-30d4-427e-a841-b3672c659203" containerName="cleanup" Jul 27 20:07:44.629582 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.629556 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:44.632686 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.632661 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xh8n\"" Jul 27 20:07:44.639238 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.639202 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224303351344"] Jul 27 20:07:44.659462 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.659423 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8bjr\" (UniqueName: \"kubernetes.io/projected/ada28f61-b36f-4fdc-a26d-d74ae445d94e-kube-api-access-j8bjr\") pod \"test-curl-7498-140224303351344\" (UID: \"ada28f61-b36f-4fdc-a26d-d74ae445d94e\") " pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:44.760449 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.760407 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8bjr\" (UniqueName: \"kubernetes.io/projected/ada28f61-b36f-4fdc-a26d-d74ae445d94e-kube-api-access-j8bjr\") pod \"test-curl-7498-140224303351344\" (UID: \"ada28f61-b36f-4fdc-a26d-d74ae445d94e\") " pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:44.780473 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.780437 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8bjr\" (UniqueName: \"kubernetes.io/projected/ada28f61-b36f-4fdc-a26d-d74ae445d94e-kube-api-access-j8bjr\") pod \"test-curl-7498-140224303351344\" (UID: \"ada28f61-b36f-4fdc-a26d-d74ae445d94e\") " pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:44.941193 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:44.941151 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:45.096612 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:45.096566 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224303351344"] Jul 27 20:07:45.098402 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:07:45.098364 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podada28f61_b36f_4fdc_a26d_d74ae445d94e.slice/crio-0e6e0feb888c2e66865dd3d390bfa9e9c0ecb281e858eacceeb3ac9fe8991d46 WatchSource:0}: Error finding container 0e6e0feb888c2e66865dd3d390bfa9e9c0ecb281e858eacceeb3ac9fe8991d46: Status 404 returned error can't find the container with id 0e6e0feb888c2e66865dd3d390bfa9e9c0ecb281e858eacceeb3ac9fe8991d46 Jul 27 20:07:45.099705 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:45.099688 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:07:45.294303 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:45.292027 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224303351344" event={"ID":"ada28f61-b36f-4fdc-a26d-d74ae445d94e","Type":"ContainerStarted","Data":"0e6e0feb888c2e66865dd3d390bfa9e9c0ecb281e858eacceeb3ac9fe8991d46"} Jul 27 20:07:46.299754 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:46.299721 2566 generic.go:356] "Generic (PLEG): container finished" podID="ada28f61-b36f-4fdc-a26d-d74ae445d94e" containerID="b9ef639646ffbc7b3f903be86e42e00864d930798afafbd5ea3d213d50eaf5c0" exitCode=0 Jul 27 20:07:46.300198 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:46.299813 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224303351344" event={"ID":"ada28f61-b36f-4fdc-a26d-d74ae445d94e","Type":"ContainerDied","Data":"b9ef639646ffbc7b3f903be86e42e00864d930798afafbd5ea3d213d50eaf5c0"} Jul 27 20:07:47.468198 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.468163 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:47.524372 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.524341 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7498-140224303351344_ada28f61-b36f-4fdc-a26d-d74ae445d94e/test-curl-7498-140224303351344/0.log" Jul 27 20:07:47.556484 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.556439 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-140224303351344"] Jul 27 20:07:47.561457 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.561421 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-140224303351344"] Jul 27 20:07:47.591069 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.590980 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8bjr\" (UniqueName: \"kubernetes.io/projected/ada28f61-b36f-4fdc-a26d-d74ae445d94e-kube-api-access-j8bjr\") pod \"ada28f61-b36f-4fdc-a26d-d74ae445d94e\" (UID: \"ada28f61-b36f-4fdc-a26d-d74ae445d94e\") " Jul 27 20:07:47.593139 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.593107 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada28f61-b36f-4fdc-a26d-d74ae445d94e-kube-api-access-j8bjr" (OuterVolumeSpecName: "kube-api-access-j8bjr") pod "ada28f61-b36f-4fdc-a26d-d74ae445d94e" (UID: "ada28f61-b36f-4fdc-a26d-d74ae445d94e"). InnerVolumeSpecName "kube-api-access-j8bjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:07:47.692178 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:47.692136 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8bjr\" (UniqueName: \"kubernetes.io/projected/ada28f61-b36f-4fdc-a26d-d74ae445d94e-kube-api-access-j8bjr\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:07:48.322144 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:48.322111 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:48.322144 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:48.322138 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e6e0feb888c2e66865dd3d390bfa9e9c0ecb281e858eacceeb3ac9fe8991d46" Jul 27 20:07:48.335940 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:07:48.335901 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-140224303351344\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="ada28f61-b36f-4fdc-a26d-d74ae445d94e" pod="opendatahub/test-curl-7498-140224303351344" Jul 27 20:07:49.547084 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:49.547039 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ada28f61-b36f-4fdc-a26d-d74ae445d94e" path="/var/lib/kubelet/pods/ada28f61-b36f-4fdc-a26d-d74ae445d94e/volumes" Jul 27 20:07:54.037173 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.037131 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-140224297900048"] Jul 27 20:07:54.037632 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.037618 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ada28f61-b36f-4fdc-a26d-d74ae445d94e" containerName="test-curl-7498-140224303351344" Jul 27 20:07:54.037632 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.037633 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ada28f61-b36f-4fdc-a26d-d74ae445d94e" containerName="test-curl-7498-140224303351344" Jul 27 20:07:54.037731 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.037720 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ada28f61-b36f-4fdc-a26d-d74ae445d94e" containerName="test-curl-7498-140224303351344" Jul 27 20:07:54.041100 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.041080 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:54.044465 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.044440 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8xh8n\"" Jul 27 20:07:54.050879 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.050847 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"] Jul 27 20:07:54.162497 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.162463 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg6c9\" (UniqueName: \"kubernetes.io/projected/804e42e7-c373-4a02-9f9b-6144c8e720ab-kube-api-access-gg6c9\") pod \"test-curl-7498-140224297900048\" (UID: \"804e42e7-c373-4a02-9f9b-6144c8e720ab\") " pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:54.263408 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.263356 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg6c9\" (UniqueName: \"kubernetes.io/projected/804e42e7-c373-4a02-9f9b-6144c8e720ab-kube-api-access-gg6c9\") pod \"test-curl-7498-140224297900048\" (UID: \"804e42e7-c373-4a02-9f9b-6144c8e720ab\") " pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:54.273254 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.273226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg6c9\" (UniqueName: \"kubernetes.io/projected/804e42e7-c373-4a02-9f9b-6144c8e720ab-kube-api-access-gg6c9\") pod \"test-curl-7498-140224297900048\" (UID: \"804e42e7-c373-4a02-9f9b-6144c8e720ab\") " pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:54.352466 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.352378 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:54.519095 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:54.519068 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"] Jul 27 20:07:54.521454 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:07:54.521414 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod804e42e7_c373_4a02_9f9b_6144c8e720ab.slice/crio-447d89d65f2c05d86d66fdfab3ec1c2bc39aa69c2276c45c9584933f09880872 WatchSource:0}: Error finding container 447d89d65f2c05d86d66fdfab3ec1c2bc39aa69c2276c45c9584933f09880872: Status 404 returned error can't find the container with id 447d89d65f2c05d86d66fdfab3ec1c2bc39aa69c2276c45c9584933f09880872 Jul 27 20:07:55.369211 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:55.369174 2566 generic.go:356] "Generic (PLEG): container finished" podID="804e42e7-c373-4a02-9f9b-6144c8e720ab" containerID="5e01002d7d86428a638c187f41533e8faf893699fb47a466ace96d6a40fedcac" exitCode=0 Jul 27 20:07:55.369585 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:55.369255 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297900048" event={"ID":"804e42e7-c373-4a02-9f9b-6144c8e720ab","Type":"ContainerDied","Data":"5e01002d7d86428a638c187f41533e8faf893699fb47a466ace96d6a40fedcac"} Jul 27 20:07:55.369585 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:55.369301 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297900048" event={"ID":"804e42e7-c373-4a02-9f9b-6144c8e720ab","Type":"ContainerStarted","Data":"447d89d65f2c05d86d66fdfab3ec1c2bc39aa69c2276c45c9584933f09880872"} Jul 27 20:07:56.534246 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.534221 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:56.557935 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.557896 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7498-140224297900048_804e42e7-c373-4a02-9f9b-6144c8e720ab/test-curl-7498-140224297900048/0.log" Jul 27 20:07:56.587077 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.587034 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"] Jul 27 20:07:56.590063 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.590027 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-140224297900048"] Jul 27 20:07:56.691489 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.691454 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gg6c9\" (UniqueName: \"kubernetes.io/projected/804e42e7-c373-4a02-9f9b-6144c8e720ab-kube-api-access-gg6c9\") pod \"804e42e7-c373-4a02-9f9b-6144c8e720ab\" (UID: \"804e42e7-c373-4a02-9f9b-6144c8e720ab\") " Jul 27 20:07:56.693602 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.693568 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804e42e7-c373-4a02-9f9b-6144c8e720ab-kube-api-access-gg6c9" (OuterVolumeSpecName: "kube-api-access-gg6c9") pod "804e42e7-c373-4a02-9f9b-6144c8e720ab" (UID: "804e42e7-c373-4a02-9f9b-6144c8e720ab"). InnerVolumeSpecName "kube-api-access-gg6c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:07:56.719402 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.719353 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-140224297901056"] Jul 27 20:07:56.719902 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.719886 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="804e42e7-c373-4a02-9f9b-6144c8e720ab" containerName="test-curl-7498-140224297900048" Jul 27 20:07:56.719959 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.719903 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="804e42e7-c373-4a02-9f9b-6144c8e720ab" containerName="test-curl-7498-140224297900048" Jul 27 20:07:56.719998 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.719981 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="804e42e7-c373-4a02-9f9b-6144c8e720ab" containerName="test-curl-7498-140224297900048" Jul 27 20:07:56.723307 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.723283 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:07:56.732501 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.732470 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"] Jul 27 20:07:56.792620 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.792586 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfrqm\" (UniqueName: \"kubernetes.io/projected/fefa9016-275d-48b6-b7eb-752c9532c812-kube-api-access-kfrqm\") pod \"test-curl-7498-140224297901056\" (UID: \"fefa9016-275d-48b6-b7eb-752c9532c812\") " pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:07:56.792851 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.792696 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg6c9\" (UniqueName: \"kubernetes.io/projected/804e42e7-c373-4a02-9f9b-6144c8e720ab-kube-api-access-gg6c9\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:07:56.894245 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.894203 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfrqm\" (UniqueName: \"kubernetes.io/projected/fefa9016-275d-48b6-b7eb-752c9532c812-kube-api-access-kfrqm\") pod \"test-curl-7498-140224297901056\" (UID: \"fefa9016-275d-48b6-b7eb-752c9532c812\") " pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:07:56.904155 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:56.904118 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfrqm\" (UniqueName: \"kubernetes.io/projected/fefa9016-275d-48b6-b7eb-752c9532c812-kube-api-access-kfrqm\") pod \"test-curl-7498-140224297901056\" (UID: \"fefa9016-275d-48b6-b7eb-752c9532c812\") " pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:07:57.035254 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:57.035150 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:07:57.211912 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:57.211883 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"] Jul 27 20:07:57.213960 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:07:57.213931 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfefa9016_275d_48b6_b7eb_752c9532c812.slice/crio-d29353a30b340f1a13d89fb8d1b5d3ebd0a5058fa6257252deed1adeba750e3c WatchSource:0}: Error finding container d29353a30b340f1a13d89fb8d1b5d3ebd0a5058fa6257252deed1adeba750e3c: Status 404 returned error can't find the container with id d29353a30b340f1a13d89fb8d1b5d3ebd0a5058fa6257252deed1adeba750e3c Jul 27 20:07:57.382847 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:57.382722 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297901056" event={"ID":"fefa9016-275d-48b6-b7eb-752c9532c812","Type":"ContainerStarted","Data":"d29353a30b340f1a13d89fb8d1b5d3ebd0a5058fa6257252deed1adeba750e3c"} Jul 27 20:07:57.384093 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:57.384063 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:57.384241 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:57.384076 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="447d89d65f2c05d86d66fdfab3ec1c2bc39aa69c2276c45c9584933f09880872" Jul 27 20:07:57.395158 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:07:57.395118 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-140224297900048\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="804e42e7-c373-4a02-9f9b-6144c8e720ab" pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:57.545018 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:07:57.544983 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-140224297900048\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="804e42e7-c373-4a02-9f9b-6144c8e720ab" pod="opendatahub/test-curl-7498-140224297900048" Jul 27 20:07:57.547266 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:57.547240 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="804e42e7-c373-4a02-9f9b-6144c8e720ab" path="/var/lib/kubelet/pods/804e42e7-c373-4a02-9f9b-6144c8e720ab/volumes" Jul 27 20:07:58.391685 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:58.391650 2566 generic.go:356] "Generic (PLEG): container finished" podID="fefa9016-275d-48b6-b7eb-752c9532c812" containerID="8f6fca6888cb96e3aeed04f437d7d955f2faeb91de09f4389365253c8de9f8be" exitCode=0 Jul 27 20:07:58.391685 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:58.391691 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-140224297901056" event={"ID":"fefa9016-275d-48b6-b7eb-752c9532c812","Type":"ContainerDied","Data":"8f6fca6888cb96e3aeed04f437d7d955f2faeb91de09f4389365253c8de9f8be"} Jul 27 20:07:59.565822 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.565789 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:07:59.589430 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.589401 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7498-140224297901056_fefa9016-275d-48b6-b7eb-752c9532c812/test-curl-7498-140224297901056/0.log" Jul 27 20:07:59.620741 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.620701 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"] Jul 27 20:07:59.624234 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.624198 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-140224297901056"] Jul 27 20:07:59.727640 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.727600 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfrqm\" (UniqueName: \"kubernetes.io/projected/fefa9016-275d-48b6-b7eb-752c9532c812-kube-api-access-kfrqm\") pod \"fefa9016-275d-48b6-b7eb-752c9532c812\" (UID: \"fefa9016-275d-48b6-b7eb-752c9532c812\") " Jul 27 20:07:59.729738 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.729709 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefa9016-275d-48b6-b7eb-752c9532c812-kube-api-access-kfrqm" (OuterVolumeSpecName: "kube-api-access-kfrqm") pod "fefa9016-275d-48b6-b7eb-752c9532c812" (UID: "fefa9016-275d-48b6-b7eb-752c9532c812"). InnerVolumeSpecName "kube-api-access-kfrqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:07:59.829314 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:07:59.829265 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfrqm\" (UniqueName: \"kubernetes.io/projected/fefa9016-275d-48b6-b7eb-752c9532c812-kube-api-access-kfrqm\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:08:00.404103 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:00.404076 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:08:00.404269 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:00.404114 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d29353a30b340f1a13d89fb8d1b5d3ebd0a5058fa6257252deed1adeba750e3c" Jul 27 20:08:00.415338 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:08:00.415303 2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7498-140224297901056\" is forbidden: User \"system:node:ip-10-0-132-244.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-244.ec2.internal' and this object" podUID="fefa9016-275d-48b6-b7eb-752c9532c812" pod="opendatahub/test-curl-7498-140224297901056" Jul 27 20:08:01.547156 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:01.547119 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fefa9016-275d-48b6-b7eb-752c9532c812" path="/var/lib/kubelet/pods/fefa9016-275d-48b6-b7eb-752c9532c812/volumes" Jul 27 20:08:28.220244 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.220201 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq"] Jul 27 20:08:28.221299 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.221273 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fefa9016-275d-48b6-b7eb-752c9532c812" containerName="test-curl-7498-140224297901056" Jul 27 20:08:28.221299 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.221296 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fefa9016-275d-48b6-b7eb-752c9532c812" containerName="test-curl-7498-140224297901056" Jul 27 20:08:28.221483 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.221391 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fefa9016-275d-48b6-b7eb-752c9532c812" containerName="test-curl-7498-140224297901056" Jul 27 20:08:28.224877 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.224847 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.227227 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.227201 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-791fb5-serving-cert\"" Jul 27 20:08:28.245399 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.245361 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq"] Jul 27 20:08:28.339075 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.339021 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dsks\" (UniqueName: \"kubernetes.io/projected/a4d9f50c-e09d-4e79-870e-3535724f4f5d-kube-api-access-6dsks\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.339274 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.339127 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.339274 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.339187 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9f50c-e09d-4e79-870e-3535724f4f5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.440654 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.440610 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dsks\" (UniqueName: \"kubernetes.io/projected/a4d9f50c-e09d-4e79-870e-3535724f4f5d-kube-api-access-6dsks\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.440887 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.440669 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.440887 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.440703 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9f50c-e09d-4e79-870e-3535724f4f5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.440887 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:08:28.440872 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-serving-cert: secret "maas-api-e2e-ipp-b-791fb5-serving-cert" not found Jul 27 20:08:28.441049 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:08:28.440937 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls podName:a4d9f50c-e09d-4e79-870e-3535724f4f5d nodeName:}" failed. No retries permitted until 2026-07-27 20:08:28.940921368 +0000 UTC m=+3081.935418321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls") pod "maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" (UID: "a4d9f50c-e09d-4e79-870e-3535724f4f5d") : secret "maas-api-e2e-ipp-b-791fb5-serving-cert" not found Jul 27 20:08:28.441352 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.441335 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9f50c-e09d-4e79-870e-3535724f4f5d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.454149 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.454116 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dsks\" (UniqueName: \"kubernetes.io/projected/a4d9f50c-e09d-4e79-870e-3535724f4f5d-kube-api-access-6dsks\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.947167 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:28.947131 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:28.947351 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:08:28.947281 2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-serving-cert: secret "maas-api-e2e-ipp-b-791fb5-serving-cert" not found Jul 27 20:08:28.947406 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:08:28.947356 2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls podName:a4d9f50c-e09d-4e79-870e-3535724f4f5d nodeName:}" failed. No retries permitted until 2026-07-27 20:08:29.947337112 +0000 UTC m=+3082.941834072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls") pod "maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" (UID: "a4d9f50c-e09d-4e79-870e-3535724f4f5d") : secret "maas-api-e2e-ipp-b-791fb5-serving-cert" not found Jul 27 20:08:29.226921 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.226815 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j"] Jul 27 20:08:29.232459 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.232425 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.239288 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.239265 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-791fb5\"" Jul 27 20:08:29.240416 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.240398 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-791fb5-dockercfg-jj4wb\"" Jul 27 20:08:29.250524 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.250496 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j"] Jul 27 20:08:29.352619 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.352585 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n42wk\" (UniqueName: \"kubernetes.io/projected/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-kube-api-access-n42wk\") pod \"payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.352619 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.352626 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.453976 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.453938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n42wk\" (UniqueName: \"kubernetes.io/projected/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-kube-api-access-n42wk\") pod \"payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.453976 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.453980 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.454728 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.454707 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.463345 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.463316 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n42wk\" (UniqueName: \"kubernetes.io/projected/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-kube-api-access-n42wk\") pod \"payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.545463 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.545363 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:29.710332 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.710294 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j"] Jul 27 20:08:29.714928 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:08:29.714867 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0835d9f0_27a1_484a_b9d5_b3ae2c0f5cb2.slice/crio-2312c81f22d9ba8f48e5859de126981c2b67b75ff0861acae631567af0e722df WatchSource:0}: Error finding container 2312c81f22d9ba8f48e5859de126981c2b67b75ff0861acae631567af0e722df: Status 404 returned error can't find the container with id 2312c81f22d9ba8f48e5859de126981c2b67b75ff0861acae631567af0e722df Jul 27 20:08:29.960116 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.960076 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:29.962639 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:29.962618 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:30.037964 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.037919 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:30.193273 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.193245 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq"] Jul 27 20:08:30.194988 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:08:30.194964 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4d9f50c_e09d_4e79_870e_3535724f4f5d.slice/crio-ac9bb253da043d09eccc4b00693a59eb82bd596772dcb510fd31bf689d9609ba WatchSource:0}: Error finding container ac9bb253da043d09eccc4b00693a59eb82bd596772dcb510fd31bf689d9609ba: Status 404 returned error can't find the container with id ac9bb253da043d09eccc4b00693a59eb82bd596772dcb510fd31bf689d9609ba Jul 27 20:08:30.594505 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.594469 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" event={"ID":"a4d9f50c-e09d-4e79-870e-3535724f4f5d","Type":"ContainerStarted","Data":"ac9bb253da043d09eccc4b00693a59eb82bd596772dcb510fd31bf689d9609ba"} Jul 27 20:08:30.596425 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.596189 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" event={"ID":"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2","Type":"ContainerStarted","Data":"d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021"} Jul 27 20:08:30.596425 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.596240 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" event={"ID":"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2","Type":"ContainerStarted","Data":"2312c81f22d9ba8f48e5859de126981c2b67b75ff0861acae631567af0e722df"} Jul 27 20:08:30.596425 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.596282 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:30.628487 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:30.628437 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" podStartSLOduration=1.628420431 podStartE2EDuration="1.628420431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:30.625409408 +0000 UTC m=+3083.619906383" watchObservedRunningTime="2026-07-27 20:08:30.628420431 +0000 UTC m=+3083.622917417" Jul 27 20:08:31.605318 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:31.605271 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" event={"ID":"a4d9f50c-e09d-4e79-870e-3535724f4f5d","Type":"ContainerStarted","Data":"ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361"} Jul 27 20:08:31.605852 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:31.605581 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:31.630003 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:31.629948 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" podStartSLOduration=3.272973376 podStartE2EDuration="3.629930295s" totalImagesPullingTime="356.956919ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:08:31.628301987 +0000 UTC m=+3084.622798958" watchObservedRunningTime="2026-07-27 20:08:31.629930295 +0000 UTC m=+3084.624427269" Jul 27 20:08:37.617557 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:37.617524 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:08:41.607708 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:41.607678 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:08:49.543510 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:49.543467 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-n54d2_21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e/payload-processing/0.log" Jul 27 20:08:49.676102 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:08:49.676071 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j_0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2/payload-processing/0.log" Jul 27 20:09:17.648620 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.648572 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jpt9h"] Jul 27 20:09:17.655112 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.655089 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:17.664142 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.664108 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jpt9h"] Jul 27 20:09:17.763651 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.763613 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kknzs\" (UniqueName: \"kubernetes.io/projected/78a601dd-ac97-4e17-a801-71d8e33ad5ea-kube-api-access-kknzs\") pod \"kuadrant-operator-catalog-jpt9h\" (UID: \"78a601dd-ac97-4e17-a801-71d8e33ad5ea\") " pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:17.864458 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.864414 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kknzs\" (UniqueName: \"kubernetes.io/projected/78a601dd-ac97-4e17-a801-71d8e33ad5ea-kube-api-access-kknzs\") pod \"kuadrant-operator-catalog-jpt9h\" (UID: \"78a601dd-ac97-4e17-a801-71d8e33ad5ea\") " pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:17.874473 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.874431 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kknzs\" (UniqueName: \"kubernetes.io/projected/78a601dd-ac97-4e17-a801-71d8e33ad5ea-kube-api-access-kknzs\") pod \"kuadrant-operator-catalog-jpt9h\" (UID: \"78a601dd-ac97-4e17-a801-71d8e33ad5ea\") " pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:17.966724 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:17.966674 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:18.127612 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:18.127583 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jpt9h"] Jul 27 20:09:18.130340 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:09:18.130303 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a601dd_ac97_4e17_a801_71d8e33ad5ea.slice/crio-c28b031b7710821cda04250dd7ce258d6e20130965e5a6c00b927adf4132c1c1 WatchSource:0}: Error finding container c28b031b7710821cda04250dd7ce258d6e20130965e5a6c00b927adf4132c1c1: Status 404 returned error can't find the container with id c28b031b7710821cda04250dd7ce258d6e20130965e5a6c00b927adf4132c1c1 Jul 27 20:09:18.904711 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:18.904678 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" event={"ID":"78a601dd-ac97-4e17-a801-71d8e33ad5ea","Type":"ContainerStarted","Data":"c28b031b7710821cda04250dd7ce258d6e20130965e5a6c00b927adf4132c1c1"} Jul 27 20:09:19.913012 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:19.912971 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" event={"ID":"78a601dd-ac97-4e17-a801-71d8e33ad5ea","Type":"ContainerStarted","Data":"f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90"} Jul 27 20:09:19.933290 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:19.933234 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" podStartSLOduration=2.149840498 podStartE2EDuration="2.933216773s" totalImagesPullingTime="783.376275ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:09:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:09:19.930106405 +0000 UTC m=+3132.924603393" watchObservedRunningTime="2026-07-27 20:09:19.933216773 +0000 UTC m=+3132.927713748" Jul 27 20:09:27.966903 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:27.966868 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:27.967342 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:27.966928 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:27.989656 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:27.989628 2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:28.990405 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:28.990375 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:29.033670 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:29.033634 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jpt9h"] Jul 27 20:09:30.979957 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:30.979911 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" podUID="78a601dd-ac97-4e17-a801-71d8e33ad5ea" containerName="registry-server" containerID="cri-o://f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90" gracePeriod=2 Jul 27 20:09:31.253749 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.253718 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:31.314497 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.314465 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kknzs\" (UniqueName: \"kubernetes.io/projected/78a601dd-ac97-4e17-a801-71d8e33ad5ea-kube-api-access-kknzs\") pod \"78a601dd-ac97-4e17-a801-71d8e33ad5ea\" (UID: \"78a601dd-ac97-4e17-a801-71d8e33ad5ea\") " Jul 27 20:09:31.316970 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.316939 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a601dd-ac97-4e17-a801-71d8e33ad5ea-kube-api-access-kknzs" (OuterVolumeSpecName: "kube-api-access-kknzs") pod "78a601dd-ac97-4e17-a801-71d8e33ad5ea" (UID: "78a601dd-ac97-4e17-a801-71d8e33ad5ea"). InnerVolumeSpecName "kube-api-access-kknzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:09:31.416069 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.416029 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kknzs\" (UniqueName: \"kubernetes.io/projected/78a601dd-ac97-4e17-a801-71d8e33ad5ea-kube-api-access-kknzs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:09:31.987118 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.987079 2566 generic.go:356] "Generic (PLEG): container finished" podID="78a601dd-ac97-4e17-a801-71d8e33ad5ea" containerID="f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90" exitCode=0 Jul 27 20:09:31.987571 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.987123 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" Jul 27 20:09:31.987571 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.987162 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" event={"ID":"78a601dd-ac97-4e17-a801-71d8e33ad5ea","Type":"ContainerDied","Data":"f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90"} Jul 27 20:09:31.987571 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.987221 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-jpt9h" event={"ID":"78a601dd-ac97-4e17-a801-71d8e33ad5ea","Type":"ContainerDied","Data":"c28b031b7710821cda04250dd7ce258d6e20130965e5a6c00b927adf4132c1c1"} Jul 27 20:09:31.987571 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.987246 2566 scope.go:117] "RemoveContainer" containerID="f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90" Jul 27 20:09:31.997503 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.997479 2566 scope.go:117] "RemoveContainer" containerID="f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90" Jul 27 20:09:31.997901 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:09:31.997880 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90\": container with ID starting with f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90 not found: ID does not exist" containerID="f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90" Jul 27 20:09:31.997958 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:31.997912 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90"} err="failed to get container status \"f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90\": rpc error: code = NotFound desc = could not find container \"f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90\": container with ID starting with f2929edb1986942b5cf46d962566447c299ddc7bb0f31922cf856d8918436d90 not found: ID does not exist" Jul 27 20:09:32.006914 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:32.006885 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jpt9h"] Jul 27 20:09:32.009407 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:32.009381 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-jpt9h"] Jul 27 20:09:33.546548 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:33.546508 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a601dd-ac97-4e17-a801-71d8e33ad5ea" path="/var/lib/kubelet/pods/78a601dd-ac97-4e17-a801-71d8e33ad5ea/volumes" Jul 27 20:09:41.784420 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:41.784388 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-n54d2_21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e/payload-processing/0.log" Jul 27 20:09:41.912192 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:41.912159 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j_0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2/payload-processing/0.log" Jul 27 20:09:46.679219 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.679181 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp"] Jul 27 20:09:46.679710 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.679693 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a601dd-ac97-4e17-a801-71d8e33ad5ea" containerName="registry-server" Jul 27 20:09:46.679710 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.679711 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a601dd-ac97-4e17-a801-71d8e33ad5ea" containerName="registry-server" Jul 27 20:09:46.679864 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.679854 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a601dd-ac97-4e17-a801-71d8e33ad5ea" containerName="registry-server" Jul 27 20:09:46.684555 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.684529 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.690691 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.690663 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-d8e48b\"" Jul 27 20:09:46.692220 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.692199 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-d8e48b-dockercfg-pk8qn\"" Jul 27 20:09:46.704947 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.704906 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp"] Jul 27 20:09:46.782853 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.782819 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kswmc\" (UniqueName: \"kubernetes.io/projected/0a916434-f08a-4ca8-9605-9c00424c9db4-kube-api-access-kswmc\") pod \"payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.783035 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.782887 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a916434-f08a-4ca8-9605-9c00424c9db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.883970 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.883922 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a916434-f08a-4ca8-9605-9c00424c9db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.884168 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.884132 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kswmc\" (UniqueName: \"kubernetes.io/projected/0a916434-f08a-4ca8-9605-9c00424c9db4-kube-api-access-kswmc\") pod \"payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.884917 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.884896 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a916434-f08a-4ca8-9605-9c00424c9db4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.897319 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.897281 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kswmc\" (UniqueName: \"kubernetes.io/projected/0a916434-f08a-4ca8-9605-9c00424c9db4-kube-api-access-kswmc\") pod \"payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:46.995336 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:46.995240 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:47.160162 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:47.160130 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp"] Jul 27 20:09:47.162628 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:09:47.162597 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a916434_f08a_4ca8_9605_9c00424c9db4.slice/crio-cd0e4d75ef780ac7aa625f3f2b60c80b24e5342d4e4071aefeeb559122ae6e9f WatchSource:0}: Error finding container cd0e4d75ef780ac7aa625f3f2b60c80b24e5342d4e4071aefeeb559122ae6e9f: Status 404 returned error can't find the container with id cd0e4d75ef780ac7aa625f3f2b60c80b24e5342d4e4071aefeeb559122ae6e9f Jul 27 20:09:48.092959 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:48.092925 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" event={"ID":"0a916434-f08a-4ca8-9605-9c00424c9db4","Type":"ContainerStarted","Data":"05ea43609f388e3dfcd210b58f56cc6c2d7c2b04902b083559d55c904d2cb337"} Jul 27 20:09:48.093385 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:48.092996 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" event={"ID":"0a916434-f08a-4ca8-9605-9c00424c9db4","Type":"ContainerStarted","Data":"cd0e4d75ef780ac7aa625f3f2b60c80b24e5342d4e4071aefeeb559122ae6e9f"} Jul 27 20:09:48.093385 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:48.093066 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:09:48.114449 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:48.114382 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" podStartSLOduration=2.114358981 podStartE2EDuration="2.114358981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:09:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:09:48.113726109 +0000 UTC m=+3161.108223086" watchObservedRunningTime="2026-07-27 20:09:48.114358981 +0000 UTC m=+3161.108855960" Jul 27 20:09:59.102125 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:09:59.102093 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:10:00.727159 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.727117 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs"] Jul 27 20:10:00.731394 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.731355 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:00.733979 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.733952 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 20:10:00.740902 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.740478 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs"] Jul 27 20:10:00.842021 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.841986 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b96ba88-eaf3-4553-8d78-615a41d176d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs\" (UID: \"5b96ba88-eaf3-4553-8d78-615a41d176d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:00.942879 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.942847 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b96ba88-eaf3-4553-8d78-615a41d176d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs\" (UID: \"5b96ba88-eaf3-4553-8d78-615a41d176d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:00.943500 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:00.943476 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b96ba88-eaf3-4553-8d78-615a41d176d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs\" (UID: \"5b96ba88-eaf3-4553-8d78-615a41d176d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:01.046287 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:01.046180 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:01.204848 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:01.204815 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs"] Jul 27 20:10:01.207048 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:10:01.207014 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b96ba88_eaf3_4553_8d78_615a41d176d6.slice/crio-c82bbc439519c677881709d568370ece68f9d15404b4020d35fe928e0bae51b7 WatchSource:0}: Error finding container c82bbc439519c677881709d568370ece68f9d15404b4020d35fe928e0bae51b7: Status 404 returned error can't find the container with id c82bbc439519c677881709d568370ece68f9d15404b4020d35fe928e0bae51b7 Jul 27 20:10:02.190456 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:02.190369 2566 generic.go:356] "Generic (PLEG): container finished" podID="5b96ba88-eaf3-4553-8d78-615a41d176d6" containerID="dd7806845cb79a667cece918862e8d765f3484f4b980bfa24c57e17ca6ec7f79" exitCode=0 Jul 27 20:10:02.190852 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:02.190453 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" event={"ID":"5b96ba88-eaf3-4553-8d78-615a41d176d6","Type":"ContainerDied","Data":"dd7806845cb79a667cece918862e8d765f3484f4b980bfa24c57e17ca6ec7f79"} Jul 27 20:10:02.190852 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:02.190500 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" event={"ID":"5b96ba88-eaf3-4553-8d78-615a41d176d6","Type":"ContainerStarted","Data":"c82bbc439519c677881709d568370ece68f9d15404b4020d35fe928e0bae51b7"} Jul 27 20:10:03.344503 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:03.344470 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:03.470395 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:03.470292 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b96ba88-eaf3-4553-8d78-615a41d176d6-maas-api-service-ca\") pod \"5b96ba88-eaf3-4553-8d78-615a41d176d6\" (UID: \"5b96ba88-eaf3-4553-8d78-615a41d176d6\") " Jul 27 20:10:03.470681 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:03.470654 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b96ba88-eaf3-4553-8d78-615a41d176d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b96ba88-eaf3-4553-8d78-615a41d176d6" (UID: "5b96ba88-eaf3-4553-8d78-615a41d176d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:10:03.571167 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:03.571130 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b96ba88-eaf3-4553-8d78-615a41d176d6-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:10:04.203349 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:04.203320 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" Jul 27 20:10:04.203549 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:04.203360 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs" event={"ID":"5b96ba88-eaf3-4553-8d78-615a41d176d6","Type":"ContainerDied","Data":"c82bbc439519c677881709d568370ece68f9d15404b4020d35fe928e0bae51b7"} Jul 27 20:10:04.203549 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:04.203414 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c82bbc439519c677881709d568370ece68f9d15404b4020d35fe928e0bae51b7" Jul 27 20:10:11.099988 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.099949 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp"] Jul 27 20:10:11.100456 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.100171 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" podUID="0a916434-f08a-4ca8-9605-9c00424c9db4" containerName="payload-pre-processing" containerID="cri-o://05ea43609f388e3dfcd210b58f56cc6c2d7c2b04902b083559d55c904d2cb337" gracePeriod=30 Jul 27 20:10:11.260387 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.260320 2566 generic.go:356] "Generic (PLEG): container finished" podID="0a916434-f08a-4ca8-9605-9c00424c9db4" containerID="05ea43609f388e3dfcd210b58f56cc6c2d7c2b04902b083559d55c904d2cb337" exitCode=0 Jul 27 20:10:11.260387 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.260366 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" event={"ID":"0a916434-f08a-4ca8-9605-9c00424c9db4","Type":"ContainerDied","Data":"05ea43609f388e3dfcd210b58f56cc6c2d7c2b04902b083559d55c904d2cb337"} Jul 27 20:10:11.400216 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.400182 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:10:11.557696 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.557666 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kswmc\" (UniqueName: \"kubernetes.io/projected/0a916434-f08a-4ca8-9605-9c00424c9db4-kube-api-access-kswmc\") pod \"0a916434-f08a-4ca8-9605-9c00424c9db4\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " Jul 27 20:10:11.557913 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.557737 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a916434-f08a-4ca8-9605-9c00424c9db4-plugins-config-volume\") pod \"0a916434-f08a-4ca8-9605-9c00424c9db4\" (UID: \"0a916434-f08a-4ca8-9605-9c00424c9db4\") " Jul 27 20:10:11.558139 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.558117 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a916434-f08a-4ca8-9605-9c00424c9db4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0a916434-f08a-4ca8-9605-9c00424c9db4" (UID: "0a916434-f08a-4ca8-9605-9c00424c9db4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:10:11.559877 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.559847 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a916434-f08a-4ca8-9605-9c00424c9db4-kube-api-access-kswmc" (OuterVolumeSpecName: "kube-api-access-kswmc") pod "0a916434-f08a-4ca8-9605-9c00424c9db4" (UID: "0a916434-f08a-4ca8-9605-9c00424c9db4"). InnerVolumeSpecName "kube-api-access-kswmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:10:11.659396 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.659287 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kswmc\" (UniqueName: \"kubernetes.io/projected/0a916434-f08a-4ca8-9605-9c00424c9db4-kube-api-access-kswmc\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:10:11.659396 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:11.659323 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a916434-f08a-4ca8-9605-9c00424c9db4-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:10:12.267823 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:12.267787 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" event={"ID":"0a916434-f08a-4ca8-9605-9c00424c9db4","Type":"ContainerDied","Data":"cd0e4d75ef780ac7aa625f3f2b60c80b24e5342d4e4071aefeeb559122ae6e9f"} Jul 27 20:10:12.267823 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:12.267818 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp" Jul 27 20:10:12.268328 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:12.267851 2566 scope.go:117] "RemoveContainer" containerID="05ea43609f388e3dfcd210b58f56cc6c2d7c2b04902b083559d55c904d2cb337" Jul 27 20:10:12.300789 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:12.300740 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp"] Jul 27 20:10:12.304508 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:12.304475 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-d8e48b-55f67f4f88-88hqp"] Jul 27 20:10:13.547566 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:13.547527 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a916434-f08a-4ca8-9605-9c00424c9db4" path="/var/lib/kubelet/pods/0a916434-f08a-4ca8-9605-9c00424c9db4/volumes" Jul 27 20:10:15.832519 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:15.832461 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs"] Jul 27 20:10:15.835473 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:15.835436 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-d8e48b-46hhs"] Jul 27 20:10:17.547069 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:17.547035 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b96ba88-eaf3-4553-8d78-615a41d176d6" path="/var/lib/kubelet/pods/5b96ba88-eaf3-4553-8d78-615a41d176d6/volumes" Jul 27 20:10:26.521427 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.521383 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr"] Jul 27 20:10:26.521944 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.521929 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b96ba88-eaf3-4553-8d78-615a41d176d6" containerName="revoke-keys" Jul 27 20:10:26.521944 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.521946 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b96ba88-eaf3-4553-8d78-615a41d176d6" containerName="revoke-keys" Jul 27 20:10:26.522044 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.521967 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a916434-f08a-4ca8-9605-9c00424c9db4" containerName="payload-pre-processing" Jul 27 20:10:26.522044 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.521973 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a916434-f08a-4ca8-9605-9c00424c9db4" containerName="payload-pre-processing" Jul 27 20:10:26.522113 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.522057 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b96ba88-eaf3-4553-8d78-615a41d176d6" containerName="revoke-keys" Jul 27 20:10:26.522113 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.522069 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a916434-f08a-4ca8-9605-9c00424c9db4" containerName="payload-pre-processing" Jul 27 20:10:26.525673 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.525651 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:26.528505 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.528475 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 20:10:26.533036 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.533008 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr"] Jul 27 20:10:26.608408 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.608371 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce8101c-6d81-432d-91ca-86cea91492b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr\" (UID: \"bce8101c-6d81-432d-91ca-86cea91492b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:26.709832 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.709753 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce8101c-6d81-432d-91ca-86cea91492b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr\" (UID: \"bce8101c-6d81-432d-91ca-86cea91492b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:26.710487 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.710457 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce8101c-6d81-432d-91ca-86cea91492b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr\" (UID: \"bce8101c-6d81-432d-91ca-86cea91492b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:26.839909 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.839809 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:26.998733 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:26.998694 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr"] Jul 27 20:10:26.999612 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:10:26.999586 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce8101c_6d81_432d_91ca_86cea91492b0.slice/crio-96ba6193e7477961e238a141c50fc78db14a58abc9a55611f0b8dd2be47673eb WatchSource:0}: Error finding container 96ba6193e7477961e238a141c50fc78db14a58abc9a55611f0b8dd2be47673eb: Status 404 returned error can't find the container with id 96ba6193e7477961e238a141c50fc78db14a58abc9a55611f0b8dd2be47673eb Jul 27 20:10:27.368205 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:27.368171 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" event={"ID":"bce8101c-6d81-432d-91ca-86cea91492b0","Type":"ContainerStarted","Data":"843f30adca27999786dc22e25f6dea067e21f77ce4ee2445fc4fabb5dd356b4c"} Jul 27 20:10:27.368409 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:27.368231 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" event={"ID":"bce8101c-6d81-432d-91ca-86cea91492b0","Type":"ContainerStarted","Data":"96ba6193e7477961e238a141c50fc78db14a58abc9a55611f0b8dd2be47673eb"} Jul 27 20:10:27.388443 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:27.388324 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" podStartSLOduration=1.388304102 podStartE2EDuration="1.388304102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:10:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:10:27.386354508 +0000 UTC m=+3200.380851484" watchObservedRunningTime="2026-07-27 20:10:27.388304102 +0000 UTC m=+3200.382801077" Jul 27 20:10:28.380453 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:28.380417 2566 generic.go:356] "Generic (PLEG): container finished" podID="bce8101c-6d81-432d-91ca-86cea91492b0" containerID="843f30adca27999786dc22e25f6dea067e21f77ce4ee2445fc4fabb5dd356b4c" exitCode=0 Jul 27 20:10:28.380866 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:28.380471 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" event={"ID":"bce8101c-6d81-432d-91ca-86cea91492b0","Type":"ContainerDied","Data":"843f30adca27999786dc22e25f6dea067e21f77ce4ee2445fc4fabb5dd356b4c"} Jul 27 20:10:29.540398 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:29.540367 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:29.642018 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:29.641917 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce8101c-6d81-432d-91ca-86cea91492b0-maas-api-service-ca\") pod \"bce8101c-6d81-432d-91ca-86cea91492b0\" (UID: \"bce8101c-6d81-432d-91ca-86cea91492b0\") " Jul 27 20:10:29.642303 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:29.642279 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce8101c-6d81-432d-91ca-86cea91492b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bce8101c-6d81-432d-91ca-86cea91492b0" (UID: "bce8101c-6d81-432d-91ca-86cea91492b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:10:29.743675 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:29.743641 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bce8101c-6d81-432d-91ca-86cea91492b0-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:10:30.394100 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:30.394070 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" Jul 27 20:10:30.394309 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:30.394108 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr" event={"ID":"bce8101c-6d81-432d-91ca-86cea91492b0","Type":"ContainerDied","Data":"96ba6193e7477961e238a141c50fc78db14a58abc9a55611f0b8dd2be47673eb"} Jul 27 20:10:30.394309 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:30.394160 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ba6193e7477961e238a141c50fc78db14a58abc9a55611f0b8dd2be47673eb" Jul 27 20:10:46.756015 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:46.755972 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr"] Jul 27 20:10:46.760220 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:46.760181 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-58c998-m24rr"] Jul 27 20:10:47.546747 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:10:47.546708 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce8101c-6d81-432d-91ca-86cea91492b0" path="/var/lib/kubelet/pods/bce8101c-6d81-432d-91ca-86cea91492b0/volumes" Jul 27 20:11:35.869590 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.869541 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9"] Jul 27 20:11:35.870128 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.870087 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce8101c-6d81-432d-91ca-86cea91492b0" containerName="revoke-keys" Jul 27 20:11:35.870128 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.870100 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce8101c-6d81-432d-91ca-86cea91492b0" containerName="revoke-keys" Jul 27 20:11:35.870202 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.870196 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce8101c-6d81-432d-91ca-86cea91492b0" containerName="revoke-keys" Jul 27 20:11:35.873717 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.873696 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:35.878049 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.878019 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 20:11:35.883999 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.883965 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9"] Jul 27 20:11:35.957360 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:35.957317 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de444d66-befc-4fd9-b644-e111457c1470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9\" (UID: \"de444d66-befc-4fd9-b644-e111457c1470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:36.058621 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.058576 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de444d66-befc-4fd9-b644-e111457c1470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9\" (UID: \"de444d66-befc-4fd9-b644-e111457c1470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:36.059415 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.059390 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de444d66-befc-4fd9-b644-e111457c1470-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9\" (UID: \"de444d66-befc-4fd9-b644-e111457c1470\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:36.186936 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.186899 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:36.335357 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.335328 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9"] Jul 27 20:11:36.337003 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:11:36.336972 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde444d66_befc_4fd9_b644_e111457c1470.slice/crio-a86d4c29685f410867469522f6f9cb22c4fd45626a681a6b5195027eb32beb54 WatchSource:0}: Error finding container a86d4c29685f410867469522f6f9cb22c4fd45626a681a6b5195027eb32beb54: Status 404 returned error can't find the container with id a86d4c29685f410867469522f6f9cb22c4fd45626a681a6b5195027eb32beb54 Jul 27 20:11:36.812546 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.812444 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" event={"ID":"de444d66-befc-4fd9-b644-e111457c1470","Type":"ContainerStarted","Data":"f78ac7abec4511f7be80439cc75c993aa470f71b3a47c759b1a2dad3c1dce3fc"} Jul 27 20:11:36.812546 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.812508 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" event={"ID":"de444d66-befc-4fd9-b644-e111457c1470","Type":"ContainerStarted","Data":"a86d4c29685f410867469522f6f9cb22c4fd45626a681a6b5195027eb32beb54"} Jul 27 20:11:36.832289 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:36.832228 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" podStartSLOduration=1.832209749 podStartE2EDuration="1.832209749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:11:36.830119277 +0000 UTC m=+3269.824616251" watchObservedRunningTime="2026-07-27 20:11:36.832209749 +0000 UTC m=+3269.826706724" Jul 27 20:11:37.821243 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:37.821197 2566 generic.go:356] "Generic (PLEG): container finished" podID="de444d66-befc-4fd9-b644-e111457c1470" containerID="f78ac7abec4511f7be80439cc75c993aa470f71b3a47c759b1a2dad3c1dce3fc" exitCode=0 Jul 27 20:11:37.821656 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:37.821273 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" event={"ID":"de444d66-befc-4fd9-b644-e111457c1470","Type":"ContainerDied","Data":"f78ac7abec4511f7be80439cc75c993aa470f71b3a47c759b1a2dad3c1dce3fc"} Jul 27 20:11:38.979725 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:38.979694 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:38.983591 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:38.983561 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de444d66-befc-4fd9-b644-e111457c1470-maas-api-service-ca\") pod \"de444d66-befc-4fd9-b644-e111457c1470\" (UID: \"de444d66-befc-4fd9-b644-e111457c1470\") " Jul 27 20:11:38.983920 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:38.983898 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de444d66-befc-4fd9-b644-e111457c1470-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de444d66-befc-4fd9-b644-e111457c1470" (UID: "de444d66-befc-4fd9-b644-e111457c1470"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:11:39.085214 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:39.085107 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de444d66-befc-4fd9-b644-e111457c1470-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:11:39.835704 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:39.835615 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" Jul 27 20:11:39.835704 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:39.835646 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9" event={"ID":"de444d66-befc-4fd9-b644-e111457c1470","Type":"ContainerDied","Data":"a86d4c29685f410867469522f6f9cb22c4fd45626a681a6b5195027eb32beb54"} Jul 27 20:11:39.835704 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:39.835694 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a86d4c29685f410867469522f6f9cb22c4fd45626a681a6b5195027eb32beb54" Jul 27 20:11:46.420140 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.420096 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq"] Jul 27 20:11:46.420724 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.420388 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" podUID="a4d9f50c-e09d-4e79-870e-3535724f4f5d" containerName="maas-api" containerID="cri-o://ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361" gracePeriod=30 Jul 27 20:11:46.475922 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.475876 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j"] Jul 27 20:11:46.476163 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.476135 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" podUID="0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" containerName="payload-processing" containerID="cri-o://d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021" gracePeriod=30 Jul 27 20:11:46.764386 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.764353 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:11:46.781024 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.780999 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:11:46.864630 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.864587 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") pod \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " Jul 27 20:11:46.864886 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.864645 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n42wk\" (UniqueName: \"kubernetes.io/projected/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-kube-api-access-n42wk\") pod \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " Jul 27 20:11:46.864886 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.864694 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-plugins-config-volume\") pod \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\" (UID: \"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2\") " Jul 27 20:11:46.864886 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.864796 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9f50c-e09d-4e79-870e-3535724f4f5d-openshift-service-ca\") pod \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " Jul 27 20:11:46.864886 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.864828 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dsks\" (UniqueName: \"kubernetes.io/projected/a4d9f50c-e09d-4e79-870e-3535724f4f5d-kube-api-access-6dsks\") pod \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\" (UID: \"a4d9f50c-e09d-4e79-870e-3535724f4f5d\") " Jul 27 20:11:46.865182 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.865152 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" (UID: "0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:11:46.865182 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.865167 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4d9f50c-e09d-4e79-870e-3535724f4f5d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a4d9f50c-e09d-4e79-870e-3535724f4f5d" (UID: "a4d9f50c-e09d-4e79-870e-3535724f4f5d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:11:46.867001 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.866976 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-kube-api-access-n42wk" (OuterVolumeSpecName: "kube-api-access-n42wk") pod "0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" (UID: "0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2"). InnerVolumeSpecName "kube-api-access-n42wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:11:46.867105 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.867074 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a4d9f50c-e09d-4e79-870e-3535724f4f5d" (UID: "a4d9f50c-e09d-4e79-870e-3535724f4f5d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 27 20:11:46.867171 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.867146 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4d9f50c-e09d-4e79-870e-3535724f4f5d-kube-api-access-6dsks" (OuterVolumeSpecName: "kube-api-access-6dsks") pod "a4d9f50c-e09d-4e79-870e-3535724f4f5d" (UID: "a4d9f50c-e09d-4e79-870e-3535724f4f5d"). InnerVolumeSpecName "kube-api-access-6dsks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:11:46.881733 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.881701 2566 generic.go:356] "Generic (PLEG): container finished" podID="a4d9f50c-e09d-4e79-870e-3535724f4f5d" containerID="ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361" exitCode=0 Jul 27 20:11:46.881917 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.881746 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" Jul 27 20:11:46.881917 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.881790 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" event={"ID":"a4d9f50c-e09d-4e79-870e-3535724f4f5d","Type":"ContainerDied","Data":"ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361"} Jul 27 20:11:46.881917 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.881839 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq" event={"ID":"a4d9f50c-e09d-4e79-870e-3535724f4f5d","Type":"ContainerDied","Data":"ac9bb253da043d09eccc4b00693a59eb82bd596772dcb510fd31bf689d9609ba"} Jul 27 20:11:46.881917 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.881863 2566 scope.go:117] "RemoveContainer" containerID="ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361" Jul 27 20:11:46.883184 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.883138 2566 generic.go:356] "Generic (PLEG): container finished" podID="0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" containerID="d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021" exitCode=0 Jul 27 20:11:46.883184 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.883184 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" Jul 27 20:11:46.883310 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.883230 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" event={"ID":"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2","Type":"ContainerDied","Data":"d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021"} Jul 27 20:11:46.883310 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.883270 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j" event={"ID":"0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2","Type":"ContainerDied","Data":"2312c81f22d9ba8f48e5859de126981c2b67b75ff0861acae631567af0e722df"} Jul 27 20:11:46.891656 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.891635 2566 scope.go:117] "RemoveContainer" containerID="ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361" Jul 27 20:11:46.892033 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:11:46.892010 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361\": container with ID starting with ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361 not found: ID does not exist" containerID="ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361" Jul 27 20:11:46.892118 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.892045 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361"} err="failed to get container status \"ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361\": rpc error: code = NotFound desc = could not find container \"ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361\": container with ID starting with ec4d0f0aad29ee987747e3005095abacc6c6bbb05761448ff0543c4593161361 not found: ID does not exist" Jul 27 20:11:46.892118 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.892068 2566 scope.go:117] "RemoveContainer" containerID="d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021" Jul 27 20:11:46.901437 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.901417 2566 scope.go:117] "RemoveContainer" containerID="d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021" Jul 27 20:11:46.901741 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:11:46.901722 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021\": container with ID starting with d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021 not found: ID does not exist" containerID="d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021" Jul 27 20:11:46.901891 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.901750 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021"} err="failed to get container status \"d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021\": rpc error: code = NotFound desc = could not find container \"d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021\": container with ID starting with d05fffdeda7fd7c708db1ac4fd6b132518778515f1519aa781cf67f941679021 not found: ID does not exist" Jul 27 20:11:46.909643 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.909589 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq"] Jul 27 20:11:46.913231 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.913195 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-791fb5-6f884497cd-7s5rq"] Jul 27 20:11:46.924067 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.924031 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j"] Jul 27 20:11:46.928357 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.928326 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-791fb5-74fb59455b-zhd6j"] Jul 27 20:11:46.966312 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.966280 2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4d9f50c-e09d-4e79-870e-3535724f4f5d-openshift-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:11:46.966312 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.966307 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dsks\" (UniqueName: \"kubernetes.io/projected/a4d9f50c-e09d-4e79-870e-3535724f4f5d-kube-api-access-6dsks\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:11:46.966312 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.966318 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4d9f50c-e09d-4e79-870e-3535724f4f5d-maas-api-tls\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:11:46.966526 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.966328 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n42wk\" (UniqueName: \"kubernetes.io/projected/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-kube-api-access-n42wk\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:11:46.966526 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:46.966336 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:11:47.547581 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:47.547546 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" path="/var/lib/kubelet/pods/0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2/volumes" Jul 27 20:11:47.548006 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:47.547992 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4d9f50c-e09d-4e79-870e-3535724f4f5d" path="/var/lib/kubelet/pods/a4d9f50c-e09d-4e79-870e-3535724f4f5d/volumes" Jul 27 20:11:50.941050 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:50.941007 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9"] Jul 27 20:11:50.945854 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:50.945820 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-791fb5-t9rc9"] Jul 27 20:11:51.547592 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:51.547559 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de444d66-befc-4fd9-b644-e111457c1470" path="/var/lib/kubelet/pods/de444d66-befc-4fd9-b644-e111457c1470/volumes" Jul 27 20:11:58.247587 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.247548 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29"] Jul 27 20:11:58.248129 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248112 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4d9f50c-e09d-4e79-870e-3535724f4f5d" containerName="maas-api" Jul 27 20:11:58.248179 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248132 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4d9f50c-e09d-4e79-870e-3535724f4f5d" containerName="maas-api" Jul 27 20:11:58.248179 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248151 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de444d66-befc-4fd9-b644-e111457c1470" containerName="revoke-keys" Jul 27 20:11:58.248179 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248157 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de444d66-befc-4fd9-b644-e111457c1470" containerName="revoke-keys" Jul 27 20:11:58.248179 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248172 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" containerName="payload-processing" Jul 27 20:11:58.248179 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248179 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" containerName="payload-processing" Jul 27 20:11:58.248343 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248257 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4d9f50c-e09d-4e79-870e-3535724f4f5d" containerName="maas-api" Jul 27 20:11:58.248343 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248266 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="de444d66-befc-4fd9-b644-e111457c1470" containerName="revoke-keys" Jul 27 20:11:58.248343 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.248275 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0835d9f0-27a1-484a-b9d5-b3ae2c0f5cb2" containerName="payload-processing" Jul 27 20:11:58.251910 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.251888 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:11:58.254337 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.254313 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rjn5q\"" Jul 27 20:11:58.260059 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.260035 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29"] Jul 27 20:11:58.381937 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.381893 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70038d35-4bdb-40db-972c-c408cf122bda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29\" (UID: \"70038d35-4bdb-40db-972c-c408cf122bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:11:58.482953 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.482908 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70038d35-4bdb-40db-972c-c408cf122bda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29\" (UID: \"70038d35-4bdb-40db-972c-c408cf122bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:11:58.483565 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.483543 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70038d35-4bdb-40db-972c-c408cf122bda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29\" (UID: \"70038d35-4bdb-40db-972c-c408cf122bda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:11:58.566394 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.566305 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:11:58.721627 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.721587 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29"] Jul 27 20:11:58.727540 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:11:58.727510 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70038d35_4bdb_40db_972c_c408cf122bda.slice/crio-32c6cd2694d645f867e8d6fefe034371d81d018e30cdc6ec7b0e89527c6f328e WatchSource:0}: Error finding container 32c6cd2694d645f867e8d6fefe034371d81d018e30cdc6ec7b0e89527c6f328e: Status 404 returned error can't find the container with id 32c6cd2694d645f867e8d6fefe034371d81d018e30cdc6ec7b0e89527c6f328e Jul 27 20:11:58.966916 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:58.966880 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" event={"ID":"70038d35-4bdb-40db-972c-c408cf122bda","Type":"ContainerStarted","Data":"32c6cd2694d645f867e8d6fefe034371d81d018e30cdc6ec7b0e89527c6f328e"} Jul 27 20:11:59.975036 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:59.974999 2566 generic.go:356] "Generic (PLEG): container finished" podID="70038d35-4bdb-40db-972c-c408cf122bda" containerID="b74ab7b39f962b42ea26641fc5224f9447260752a4d6912aa605f5480d753b5b" exitCode=0 Jul 27 20:11:59.975461 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:11:59.975069 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" event={"ID":"70038d35-4bdb-40db-972c-c408cf122bda","Type":"ContainerDied","Data":"b74ab7b39f962b42ea26641fc5224f9447260752a4d6912aa605f5480d753b5b"} Jul 27 20:12:01.139349 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.139319 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:12:01.211867 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.211825 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70038d35-4bdb-40db-972c-c408cf122bda-maas-api-service-ca\") pod \"70038d35-4bdb-40db-972c-c408cf122bda\" (UID: \"70038d35-4bdb-40db-972c-c408cf122bda\") " Jul 27 20:12:01.212252 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.212231 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70038d35-4bdb-40db-972c-c408cf122bda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "70038d35-4bdb-40db-972c-c408cf122bda" (UID: "70038d35-4bdb-40db-972c-c408cf122bda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:01.313216 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.313118 2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70038d35-4bdb-40db-972c-c408cf122bda-maas-api-service-ca\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:01.988026 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.987998 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" Jul 27 20:12:01.988216 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.987997 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29" event={"ID":"70038d35-4bdb-40db-972c-c408cf122bda","Type":"ContainerDied","Data":"32c6cd2694d645f867e8d6fefe034371d81d018e30cdc6ec7b0e89527c6f328e"} Jul 27 20:12:01.988216 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:01.988127 2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c6cd2694d645f867e8d6fefe034371d81d018e30cdc6ec7b0e89527c6f328e" Jul 27 20:12:07.871553 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:07.871520 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:12:07.882079 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:07.882052 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:12:08.803083 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:08.803009 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n"] Jul 27 20:12:08.806202 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:08.806170 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-4289a9-29753040-2gl8n"] Jul 27 20:12:09.546836 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:09.546797 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71c5dbe-30d4-427e-a841-b3672c659203" path="/var/lib/kubelet/pods/f71c5dbe-30d4-427e-a841-b3672c659203/volumes" Jul 27 20:12:13.330191 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:13.330153 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29"] Jul 27 20:12:13.337054 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:13.337016 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-4289a9-lgz29"] Jul 27 20:12:13.547010 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:13.546973 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70038d35-4bdb-40db-972c-c408cf122bda" path="/var/lib/kubelet/pods/70038d35-4bdb-40db-972c-c408cf122bda/volumes" Jul 27 20:12:20.403816 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:20.403267 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg"] Jul 27 20:12:20.403816 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:20.403557 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" containerID="cri-o://259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669" gracePeriod=30 Jul 27 20:12:25.684637 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.684564 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 20:12:25.758905 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.758871 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-data\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759082 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.758925 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istiod-ca-cert\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759082 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.758967 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x2rs\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-kube-api-access-6x2rs\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759082 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759018 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-certs\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759082 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759033 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-envoy\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759082 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759062 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-token\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759355 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759135 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-socket\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759355 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759177 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-credential-socket\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759355 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759208 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-podinfo\") pod \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\" (UID: \"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696\") " Jul 27 20:12:25.759355 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759308 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:25.759582 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759459 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:25.759582 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759478 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:25.759691 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759647 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:25.759691 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759663 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.759691 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759680 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.759869 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759693 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.759869 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.759755 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:25.761310 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.761287 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 20:12:25.761432 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.761313 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-kube-api-access-6x2rs" (OuterVolumeSpecName: "kube-api-access-6x2rs") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "kube-api-access-6x2rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:25.761432 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.761329 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:25.761511 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.761445 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" (UID: "cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:25.861169 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.861125 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.861169 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.861160 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.861169 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.861169 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.861429 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.861198 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x2rs\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-kube-api-access-6x2rs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.861429 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.861207 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:25.861429 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:25.861216 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:26.146797 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.146681 2566 generic.go:356] "Generic (PLEG): container finished" podID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerID="259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669" exitCode=0 Jul 27 20:12:26.146797 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.146744 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" Jul 27 20:12:26.146987 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.146793 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" event={"ID":"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696","Type":"ContainerDied","Data":"259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669"} Jul 27 20:12:26.146987 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.146877 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg" event={"ID":"cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696","Type":"ContainerDied","Data":"031e47aa6d13e67eb6cff9fd8844b6ae15d4c6c3dd11778850214773eb79f4a5"} Jul 27 20:12:26.146987 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.146901 2566 scope.go:117] "RemoveContainer" containerID="259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669" Jul 27 20:12:26.157489 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.157470 2566 scope.go:117] "RemoveContainer" containerID="8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce" Jul 27 20:12:26.167135 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.167071 2566 scope.go:117] "RemoveContainer" containerID="259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669" Jul 27 20:12:26.167491 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:12:26.167461 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669\": container with ID starting with 259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669 not found: ID does not exist" containerID="259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669" Jul 27 20:12:26.167575 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.167500 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669"} err="failed to get container status \"259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669\": rpc error: code = NotFound desc = could not find container \"259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669\": container with ID starting with 259335748f91e8ddfcdd95cf9f03c77872d6214e781774ba1e58cadd29ddc669 not found: ID does not exist" Jul 27 20:12:26.167575 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.167521 2566 scope.go:117] "RemoveContainer" containerID="8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce" Jul 27 20:12:26.167872 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:12:26.167848 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce\": container with ID starting with 8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce not found: ID does not exist" containerID="8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce" Jul 27 20:12:26.167951 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.167878 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce"} err="failed to get container status \"8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce\": rpc error: code = NotFound desc = could not find container \"8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce\": container with ID starting with 8994848d62d6b80853b03e421d25690e728f616402205a2ffcb5f112e737e2ce not found: ID does not exist" Jul 27 20:12:26.169875 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.169855 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg"] Jul 27 20:12:26.176210 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:26.176182 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-4289a9-openshift-default-7f556c69b-2hxsg"] Jul 27 20:12:27.548175 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:27.548141 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" path="/var/lib/kubelet/pods/cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696/volumes" Jul 27 20:12:31.413427 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.413305 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r"] Jul 27 20:12:31.413957 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.413566 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" podUID="11cabc00-a25e-4bbc-886f-34721cdc8f91" containerName="payload-processing" containerID="cri-o://63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306" gracePeriod=30 Jul 27 20:12:31.702794 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.702750 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 20:12:31.824636 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.824590 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxn79\" (UniqueName: \"kubernetes.io/projected/11cabc00-a25e-4bbc-886f-34721cdc8f91-kube-api-access-bxn79\") pod \"11cabc00-a25e-4bbc-886f-34721cdc8f91\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " Jul 27 20:12:31.824853 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.824716 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cabc00-a25e-4bbc-886f-34721cdc8f91-plugins-config-volume\") pod \"11cabc00-a25e-4bbc-886f-34721cdc8f91\" (UID: \"11cabc00-a25e-4bbc-886f-34721cdc8f91\") " Jul 27 20:12:31.825129 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.825103 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cabc00-a25e-4bbc-886f-34721cdc8f91-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "11cabc00-a25e-4bbc-886f-34721cdc8f91" (UID: "11cabc00-a25e-4bbc-886f-34721cdc8f91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:31.826897 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.826872 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11cabc00-a25e-4bbc-886f-34721cdc8f91-kube-api-access-bxn79" (OuterVolumeSpecName: "kube-api-access-bxn79") pod "11cabc00-a25e-4bbc-886f-34721cdc8f91" (UID: "11cabc00-a25e-4bbc-886f-34721cdc8f91"). InnerVolumeSpecName "kube-api-access-bxn79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:31.926156 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.926109 2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/11cabc00-a25e-4bbc-886f-34721cdc8f91-plugins-config-volume\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:31.926156 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:31.926146 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxn79\" (UniqueName: \"kubernetes.io/projected/11cabc00-a25e-4bbc-886f-34721cdc8f91-kube-api-access-bxn79\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:32.186039 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.186001 2566 generic.go:356] "Generic (PLEG): container finished" podID="11cabc00-a25e-4bbc-886f-34721cdc8f91" containerID="63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306" exitCode=0 Jul 27 20:12:32.186240 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.186049 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" Jul 27 20:12:32.186240 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.186090 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" event={"ID":"11cabc00-a25e-4bbc-886f-34721cdc8f91","Type":"ContainerDied","Data":"63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306"} Jul 27 20:12:32.186240 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.186145 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r" event={"ID":"11cabc00-a25e-4bbc-886f-34721cdc8f91","Type":"ContainerDied","Data":"372261dd0e01ea36c343cf3cccc93751ddbcc1cdac94f8674c8c3f5d2e0cfc78"} Jul 27 20:12:32.186240 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.186158 2566 scope.go:117] "RemoveContainer" containerID="63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306" Jul 27 20:12:32.195914 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.195885 2566 scope.go:117] "RemoveContainer" containerID="63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306" Jul 27 20:12:32.196279 ip-10-0-132-244 kubenswrapper[2566]: E0727 20:12:32.196257 2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306\": container with ID starting with 63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306 not found: ID does not exist" containerID="63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306" Jul 27 20:12:32.196369 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.196291 2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306"} err="failed to get container status \"63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306\": rpc error: code = NotFound desc = could not find container \"63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306\": container with ID starting with 63212be77b2cd1cf80e32b7a09da3eee15bc6ee9d43d58057bb458d8786c6306 not found: ID does not exist" Jul 27 20:12:32.209129 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.209056 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r"] Jul 27 20:12:32.212887 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:32.212851 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-f99bfd-c67cbdc94-zf72r"] Jul 27 20:12:33.546875 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:33.546843 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cabc00-a25e-4bbc-886f-34721cdc8f91" path="/var/lib/kubelet/pods/11cabc00-a25e-4bbc-886f-34721cdc8f91/volumes" Jul 27 20:12:43.154837 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:43.154793 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6"] Jul 27 20:12:43.155446 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:43.155033 2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" containerID="cri-o://5af8e854b00949fd154a7c139b6a4f7ff091491847547952d2f590f475509c93" gracePeriod=30 Jul 27 20:12:48.287938 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.287905 2566 generic.go:356] "Generic (PLEG): container finished" podID="e646924b-5519-419a-96b9-07f77f0a132c" containerID="5af8e854b00949fd154a7c139b6a4f7ff091491847547952d2f590f475509c93" exitCode=0 Jul 27 20:12:48.288397 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.287981 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" event={"ID":"e646924b-5519-419a-96b9-07f77f0a132c","Type":"ContainerDied","Data":"5af8e854b00949fd154a7c139b6a4f7ff091491847547952d2f590f475509c93"} Jul 27 20:12:48.288397 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.288069 2566 scope.go:117] "RemoveContainer" containerID="44307ceb87fc7b371221c162c0af4b6aa4baf6f166ec9fe527037046fd6df6d7" Jul 27 20:12:48.440299 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.440271 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 20:12:48.599127 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599083 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e646924b-5519-419a-96b9-07f77f0a132c-istio-podinfo\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599310 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599138 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-istio-token\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599310 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599194 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-credential-socket\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599310 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599293 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-socket\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599447 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599342 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gfltq\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-kube-api-access-gfltq\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599447 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599398 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e646924b-5519-419a-96b9-07f77f0a132c-istiod-ca-cert\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599447 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599434 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-certs\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599600 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599449 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:48.599600 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599504 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-data\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599600 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599555 2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-envoy\") pod \"e646924b-5519-419a-96b9-07f77f0a132c\" (UID: \"e646924b-5519-419a-96b9-07f77f0a132c\") " Jul 27 20:12:48.599797 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599667 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:48.599797 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599678 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:48.599923 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.599863 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e646924b-5519-419a-96b9-07f77f0a132c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 27 20:12:48.600059 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.600031 2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-certs\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.600059 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.600057 2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-credential-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.600255 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.600074 2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-workload-socket\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.600255 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.600087 2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e646924b-5519-419a-96b9-07f77f0a132c-istiod-ca-cert\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.600520 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.600484 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-data" (OuterVolumeSpecName: "istio-data") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:48.601712 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.601675 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-kube-api-access-gfltq" (OuterVolumeSpecName: "kube-api-access-gfltq") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "kube-api-access-gfltq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:48.601936 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.601777 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 27 20:12:48.601936 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.601853 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e646924b-5519-419a-96b9-07f77f0a132c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 27 20:12:48.602103 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.602085 2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-istio-token" (OuterVolumeSpecName: "istio-token") pod "e646924b-5519-419a-96b9-07f77f0a132c" (UID: "e646924b-5519-419a-96b9-07f77f0a132c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 27 20:12:48.701155 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.701115 2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfltq\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-kube-api-access-gfltq\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.701155 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.701150 2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-data\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.701155 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.701164 2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e646924b-5519-419a-96b9-07f77f0a132c-istio-envoy\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.701406 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.701175 2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e646924b-5519-419a-96b9-07f77f0a132c-istio-podinfo\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:48.701406 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:48.701186 2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e646924b-5519-419a-96b9-07f77f0a132c-istio-token\") on node \"ip-10-0-132-244.ec2.internal\" DevicePath \"\"" Jul 27 20:12:49.174678 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.174645 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kb4qg_248dea2e-c215-43ba-b4a7-bde0c0581a2d/manager/0.log" Jul 27 20:12:49.293671 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.293640 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bj7cf_f63021b5-abd9-47ce-95f8-540a45408c39/manager/0.log" Jul 27 20:12:49.295637 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.295600 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" event={"ID":"e646924b-5519-419a-96b9-07f77f0a132c","Type":"ContainerDied","Data":"441e85999cf9572012a111c53b777a0102468cb3cf1468e280525b04d5a46ba7"} Jul 27 20:12:49.295801 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.295664 2566 scope.go:117] "RemoveContainer" containerID="5af8e854b00949fd154a7c139b6a4f7ff091491847547952d2f590f475509c93" Jul 27 20:12:49.295801 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.295617 2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6" Jul 27 20:12:49.321749 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.321713 2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6"] Jul 27 20:12:49.328661 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.328626 2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-f99bfd-openshift-default-6898796cf6-cqqw6"] Jul 27 20:12:49.415643 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.415612 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f96b9564b-njdhs_0749ba4f-015e-4d0f-bc0d-dd26067f3140/manager/0.log" Jul 27 20:12:49.520190 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.520116 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r7wzv_fa666236-ec46-48f4-af3c-32ce4e76e385/server/0.log" Jul 27 20:12:49.546647 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.546613 2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e646924b-5519-419a-96b9-07f77f0a132c" path="/var/lib/kubelet/pods/e646924b-5519-419a-96b9-07f77f0a132c/volumes" Jul 27 20:12:49.633639 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.633606 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hkcrl_adacd43a-7371-4790-ae78-409decc5623b/manager/0.log" Jul 27 20:12:49.863055 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.862973 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-qml79_dc902e20-17ff-4d29-80fa-5acf84729993/copy-manifests/0.log" Jul 27 20:12:49.870141 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:49.870109 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-qml79_dc902e20-17ff-4d29-80fa-5acf84729993/manager/0.log" Jul 27 20:12:50.366997 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:50.366958 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-76dd68f85-5fgqq_e2eaedbb-efe0-485a-9c29-70e68ff63b26/maas-api/0.log" Jul 27 20:12:50.480170 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:50.480137 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753010-qwqhv_4e808457-6f72-4140-acc5-f5ceea91d619/cleanup/0.log" Jul 27 20:12:50.597774 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:50.597733 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753025-d6tcd_400e38ae-bcd8-4f22-89a7-ca1eaf9694ed/cleanup/0.log" Jul 27 20:12:50.715039 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:50.715011 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29753040-87ndn_f2310426-2c7f-4d4e-8ba2-4a8be36174be/cleanup/0.log" Jul 27 20:12:50.830733 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:50.830695 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-cvxln_5a1018fe-445d-4793-bfb2-15565b6b0f77/postgres/0.log" Jul 27 20:12:52.225263 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:52.225226 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-prpnn_ff042dda-701f-4148-96fd-ab5e1e04d6b2/manager/0.log" Jul 27 20:12:52.333997 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:52.333963 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5wn9r_f41b4a46-95f2-4430-a050-1742e1f38180/manager/0.log" Jul 27 20:12:52.441317 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:52.441286 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-k9wlh_ee52f17c-b2db-4b48-9058-28f205a9de33/kuadrant-console-plugin/0.log" Jul 27 20:12:52.553666 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:52.553569 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rm2np_d46166c0-1a68-45b5-ad0c-b622351828a8/registry-server/0.log" Jul 27 20:12:52.669999 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:52.669967 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-z299b_db6abb4d-8ae3-4324-bc0c-f542c6d4d743/manager/0.log" Jul 27 20:12:53.237960 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:53.237914 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q_4d3437a2-8940-4738-8105-dd572305c90e/istio-proxy/0.log" Jul 27 20:12:53.566403 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:53.566321 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nqzgw_ddd49638-fe81-4f70-aa89-3858efdfacb2/kube-auth-proxy/0.log" Jul 27 20:12:53.680401 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:53.680368 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/1.log" Jul 27 20:12:53.793672 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:53.793639 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-djt9q_aeb8f814-39a2-4e03-9b40-5c6ab3981874/payload-pre-processing/0.log" Jul 27 20:12:53.909314 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:53.909225 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-n54d2_21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e/payload-processing/0.log" Jul 27 20:12:54.020583 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:54.020555 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85766f5879-5lm2d_6355fff2-3591-45c4-a7e3-d30721cf1b69/router/0.log" Jul 27 20:12:54.356671 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:54.356640 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-qmg96_7689531c-be6e-428c-afd0-7d81695ebaa7/main/0.log" Jul 27 20:12:59.980400 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980358 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b4zt/must-gather-lp66t"] Jul 27 20:12:59.980899 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980883 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" Jul 27 20:12:59.980950 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980900 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" Jul 27 20:12:59.980950 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980911 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70038d35-4bdb-40db-972c-c408cf122bda" containerName="revoke-keys" Jul 27 20:12:59.980950 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980920 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70038d35-4bdb-40db-972c-c408cf122bda" containerName="revoke-keys" Jul 27 20:12:59.980950 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980939 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" Jul 27 20:12:59.980950 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980946 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980953 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980958 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980967 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cabc00-a25e-4bbc-886f-34721cdc8f91" containerName="payload-processing" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980973 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cabc00-a25e-4bbc-886f-34721cdc8f91" containerName="payload-processing" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980981 2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.980989 2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.981062 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.981073 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cabc00-a25e-4bbc-886f-34721cdc8f91" containerName="payload-processing" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.981082 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70038d35-4bdb-40db-972c-c408cf122bda" containerName="revoke-keys" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.981089 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" Jul 27 20:12:59.981104 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.981097 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e646924b-5519-419a-96b9-07f77f0a132c" containerName="istio-proxy" Jul 27 20:12:59.981438 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.981242 2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf7c42ae-fa6f-41d7-8e49-c1ea86ba1696" containerName="istio-proxy" Jul 27 20:12:59.984697 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.984675 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:12:59.987906 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.987878 2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5b4zt\"/\"default-dockercfg-m29w6\"" Jul 27 20:12:59.988168 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.988148 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b4zt\"/\"kube-root-ca.crt\"" Jul 27 20:12:59.988215 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:12:59.988174 2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5b4zt\"/\"openshift-service-ca.crt\"" Jul 27 20:13:00.003187 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.003147 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvwp8\" (UniqueName: \"kubernetes.io/projected/5a8a9cb6-d76d-4fb8-a884-c903170607b5-kube-api-access-rvwp8\") pod \"must-gather-lp66t\" (UID: \"5a8a9cb6-d76d-4fb8-a884-c903170607b5\") " pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.003355 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.003212 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5a8a9cb6-d76d-4fb8-a884-c903170607b5-must-gather-output\") pod \"must-gather-lp66t\" (UID: \"5a8a9cb6-d76d-4fb8-a884-c903170607b5\") " pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.023507 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.023456 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b4zt/must-gather-lp66t"] Jul 27 20:13:00.103937 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.103898 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvwp8\" (UniqueName: \"kubernetes.io/projected/5a8a9cb6-d76d-4fb8-a884-c903170607b5-kube-api-access-rvwp8\") pod \"must-gather-lp66t\" (UID: \"5a8a9cb6-d76d-4fb8-a884-c903170607b5\") " pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.103937 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.103938 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5a8a9cb6-d76d-4fb8-a884-c903170607b5-must-gather-output\") pod \"must-gather-lp66t\" (UID: \"5a8a9cb6-d76d-4fb8-a884-c903170607b5\") " pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.104241 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.104226 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/5a8a9cb6-d76d-4fb8-a884-c903170607b5-must-gather-output\") pod \"must-gather-lp66t\" (UID: \"5a8a9cb6-d76d-4fb8-a884-c903170607b5\") " pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.112664 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.112635 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvwp8\" (UniqueName: \"kubernetes.io/projected/5a8a9cb6-d76d-4fb8-a884-c903170607b5-kube-api-access-rvwp8\") pod \"must-gather-lp66t\" (UID: \"5a8a9cb6-d76d-4fb8-a884-c903170607b5\") " pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.294978 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.294882 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b4zt/must-gather-lp66t" Jul 27 20:13:00.449443 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.449416 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b4zt/must-gather-lp66t"] Jul 27 20:13:00.451559 ip-10-0-132-244 kubenswrapper[2566]: W0727 20:13:00.451532 2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a8a9cb6_d76d_4fb8_a884_c903170607b5.slice/crio-05ca19a65d5e7c595007c6dc5a9206ba30c3a6abec8e6b663d173aaac7332ccc WatchSource:0}: Error finding container 05ca19a65d5e7c595007c6dc5a9206ba30c3a6abec8e6b663d173aaac7332ccc: Status 404 returned error can't find the container with id 05ca19a65d5e7c595007c6dc5a9206ba30c3a6abec8e6b663d173aaac7332ccc Jul 27 20:13:00.453303 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:00.453286 2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 27 20:13:01.369323 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:01.369263 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/must-gather-lp66t" event={"ID":"5a8a9cb6-d76d-4fb8-a884-c903170607b5","Type":"ContainerStarted","Data":"05ca19a65d5e7c595007c6dc5a9206ba30c3a6abec8e6b663d173aaac7332ccc"} Jul 27 20:13:02.381748 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:02.381712 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/must-gather-lp66t" event={"ID":"5a8a9cb6-d76d-4fb8-a884-c903170607b5","Type":"ContainerStarted","Data":"4ee90d7de6faa33fe217d93a6a3bb604430d48575edf01d7d9432af969f246c2"} Jul 27 20:13:02.382332 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:02.381785 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/must-gather-lp66t" event={"ID":"5a8a9cb6-d76d-4fb8-a884-c903170607b5","Type":"ContainerStarted","Data":"ffaaa1e3ed2bc6c6264ac5c9265b22007cdc58cd2c6c07844081718a82686388"} Jul 27 20:13:02.406731 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:02.406656 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b4zt/must-gather-lp66t" podStartSLOduration=2.567846449 podStartE2EDuration="3.406631052s" totalImagesPullingTime="838.784603ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-27 20:12:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:02.401467347 +0000 UTC m=+3355.395964321" watchObservedRunningTime="2026-07-27 20:13:02.406631052 +0000 UTC m=+3355.401128030" Jul 27 20:13:03.259221 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:03.259184 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-vnw5n_5bfae84b-f178-4938-a7db-12bc33b72023/global-pull-secret-syncer/0.log" Jul 27 20:13:03.405815 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:03.405779 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qn58b_48a0d645-1915-42a5-8aea-648f94ea0190/konnectivity-agent/0.log" Jul 27 20:13:03.461751 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:03.461715 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-244.ec2.internal_4dbb63acc39cffb9179e67a48cdbfda3/haproxy/0.log" Jul 27 20:13:08.135417 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:08.135360 2566 scope.go:117] "RemoveContainer" containerID="2e857fa939e20cf526efc7012bb014f6141fe857e891e6c86dba92a7035ca33d" Jul 27 20:13:09.202475 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:09.202439 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-prpnn_ff042dda-701f-4148-96fd-ab5e1e04d6b2/manager/0.log" Jul 27 20:13:09.232754 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:09.232713 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5wn9r_f41b4a46-95f2-4430-a050-1742e1f38180/manager/0.log" Jul 27 20:13:09.271375 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:09.271330 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-k9wlh_ee52f17c-b2db-4b48-9058-28f205a9de33/kuadrant-console-plugin/0.log" Jul 27 20:13:09.326666 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:09.326632 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-rm2np_d46166c0-1a68-45b5-ad0c-b622351828a8/registry-server/0.log" Jul 27 20:13:09.584852 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:09.584307 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-z299b_db6abb4d-8ae3-4324-bc0c-f542c6d4d743/manager/0.log" Jul 27 20:13:11.393860 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.393829 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/alertmanager/0.log" Jul 27 20:13:11.421917 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.421885 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/config-reloader/0.log" Jul 27 20:13:11.447373 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.447333 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/kube-rbac-proxy-web/0.log" Jul 27 20:13:11.473246 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.473211 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/kube-rbac-proxy/0.log" Jul 27 20:13:11.500785 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.500723 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/kube-rbac-proxy-metric/0.log" Jul 27 20:13:11.532208 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.532175 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/prom-label-proxy/0.log" Jul 27 20:13:11.562320 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.562286 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_93d94812-6114-4d0c-a730-0da27e8c97eb/init-config-reloader/0.log" Jul 27 20:13:11.615019 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.614978 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-5zpv4_0232b815-0718-47fb-b4cb-7b709ac7bcdf/cluster-monitoring-operator/0.log" Jul 27 20:13:11.644192 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.644115 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cltnn_b19c24bb-275c-4170-8d55-28a215b870ea/kube-state-metrics/0.log" Jul 27 20:13:11.663413 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.663375 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cltnn_b19c24bb-275c-4170-8d55-28a215b870ea/kube-rbac-proxy-main/0.log" Jul 27 20:13:11.695738 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.695702 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-cltnn_b19c24bb-275c-4170-8d55-28a215b870ea/kube-rbac-proxy-self/0.log" Jul 27 20:13:11.793779 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.793734 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jk2tq_7b509d8e-af00-442e-a7ea-3c25bbd4c7c5/node-exporter/0.log" Jul 27 20:13:11.820002 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.819926 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jk2tq_7b509d8e-af00-442e-a7ea-3c25bbd4c7c5/kube-rbac-proxy/0.log" Jul 27 20:13:11.843411 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:11.843386 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jk2tq_7b509d8e-af00-442e-a7ea-3c25bbd4c7c5/init-textfile/0.log" Jul 27 20:13:12.024731 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:12.024700 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5fjrl_9de62a0d-47ff-4f0e-9d87-88a50c813191/kube-rbac-proxy-main/0.log" Jul 27 20:13:12.046709 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:12.046682 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5fjrl_9de62a0d-47ff-4f0e-9d87-88a50c813191/kube-rbac-proxy-self/0.log" Jul 27 20:13:12.069648 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:12.069615 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-5fjrl_9de62a0d-47ff-4f0e-9d87-88a50c813191/openshift-state-metrics/0.log" Jul 27 20:13:12.347679 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:12.347597 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84c5c8b468-tdcsk_4d22b0b9-363a-4de2-aa06-6ddddf4bae90/telemeter-client/0.log" Jul 27 20:13:12.370224 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:12.370194 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84c5c8b468-tdcsk_4d22b0b9-363a-4de2-aa06-6ddddf4bae90/reload/0.log" Jul 27 20:13:12.392457 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:12.392424 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-84c5c8b468-tdcsk_4d22b0b9-363a-4de2-aa06-6ddddf4bae90/kube-rbac-proxy/0.log" Jul 27 20:13:13.198723 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.198683 2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs"] Jul 27 20:13:13.207634 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.207599 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.213211 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.213167 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs"] Jul 27 20:13:13.259887 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.259845 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-sys\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.260076 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.259957 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-lib-modules\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.260076 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.260043 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-podres\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.260177 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.260088 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmmkv\" (UniqueName: \"kubernetes.io/projected/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-kube-api-access-pmmkv\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.260234 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.260181 2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-proc\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361113 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361070 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-sys\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361326 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361131 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-lib-modules\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361326 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361165 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-podres\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361326 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361201 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmmkv\" (UniqueName: \"kubernetes.io/projected/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-kube-api-access-pmmkv\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361326 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361214 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-sys\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361326 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361308 2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-proc\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361609 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361417 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-proc\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361609 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361537 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-lib-modules\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.361609 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.361584 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-podres\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.370531 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.370498 2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmmkv\" (UniqueName: \"kubernetes.io/projected/a12a4b61-0a59-4b3b-bb41-57a6a8be71b8-kube-api-access-pmmkv\") pod \"perf-node-gather-daemonset-kxjcs\" (UID: \"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8\") " pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.524201 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.524093 2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:13.738032 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.737998 2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs"] Jul 27 20:13:13.790455 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:13.790372 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-nz774_0d1804d3-68c8-4a5e-acdd-cd30a74ee880/networking-console-plugin/0.log" Jul 27 20:13:14.472406 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:14.472369 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/1.log" Jul 27 20:13:14.478822 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:14.478786 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-bj8bq_4b2df6ab-5abe-407a-a849-f0b37f84e426/console-operator/2.log" Jul 27 20:13:14.519245 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:14.519202 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" event={"ID":"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8","Type":"ContainerStarted","Data":"8dfa68a1da1c5744364279c60fc5981ffb672555786382676d0210a6207715b0"} Jul 27 20:13:14.519468 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:14.519277 2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" event={"ID":"a12a4b61-0a59-4b3b-bb41-57a6a8be71b8","Type":"ContainerStarted","Data":"7b30157efcc209ebab96e76ecfd4f595f6fefbaea40dc399d7901b9ff0117ff4"} Jul 27 20:13:14.520257 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:14.520227 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:14.541091 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:14.541036 2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" podStartSLOduration=1.541017898 podStartE2EDuration="1.541017898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-27 20:13:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-27 20:13:14.54063881 +0000 UTC m=+3367.535135786" watchObservedRunningTime="2026-07-27 20:13:14.541017898 +0000 UTC m=+3367.535514926" Jul 27 20:13:15.077242 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:15.077205 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7cbccf9876-mtq5l_90819a44-c71e-4337-89cf-ee6993332b65/console/0.log" Jul 27 20:13:15.849102 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:15.849074 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-v8tnr_febd5129-bfda-4f84-93cd-a00c74925745/volume-data-source-validator/0.log" Jul 27 20:13:16.972999 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:16.972971 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jbsfn_004999e4-6617-4cd9-9d50-f57ad32a8d42/dns/0.log" Jul 27 20:13:16.997242 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:16.997211 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jbsfn_004999e4-6617-4cd9-9d50-f57ad32a8d42/kube-rbac-proxy/0.log" Jul 27 20:13:17.093379 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:17.093352 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4wrcm_57f91ac2-c9b3-4a51-a772-65ec3dc4e0c4/dns-node-resolver/0.log" Jul 27 20:13:17.870214 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:17.870174 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hnnwt_18f93f18-ce46-4be6-8870-9bb9db422b11/node-ca/0.log" Jul 27 20:13:18.952139 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:18.952105 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q_4d3437a2-8940-4738-8105-dd572305c90e/istio-proxy/0.log" Jul 27 20:13:19.156219 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:19.156187 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nqzgw_ddd49638-fe81-4f70-aa89-3858efdfacb2/kube-auth-proxy/0.log" Jul 27 20:13:19.189112 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:19.189082 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/0.log" Jul 27 20:13:19.223291 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:19.223135 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/1.log" Jul 27 20:13:19.257997 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:19.257957 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-djt9q_aeb8f814-39a2-4e03-9b40-5c6ab3981874/payload-pre-processing/0.log" Jul 27 20:13:19.289737 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:19.289701 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-n54d2_21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e/payload-processing/0.log" Jul 27 20:13:19.318683 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:19.318652 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85766f5879-5lm2d_6355fff2-3591-45c4-a7e3-d30721cf1b69/router/0.log" Jul 27 20:13:20.142420 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:20.142362 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-4ljv4_faa6c9de-d2e1-4800-8057-e7f07488fbe3/serve-healthcheck-canary/0.log" Jul 27 20:13:20.707270 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:20.707244 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-c8w4q_2bfb6007-ae94-4c6c-814f-eead25c3af21/insights-operator/0.log" Jul 27 20:13:20.708434 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:20.708383 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-c8w4q_2bfb6007-ae94-4c6c-814f-eead25c3af21/insights-operator/1.log" Jul 27 20:13:20.876494 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:20.876466 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qmvch_18301c03-eb72-4b76-ade3-cdb912c0b462/kube-rbac-proxy/0.log" Jul 27 20:13:20.901724 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:20.901696 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qmvch_18301c03-eb72-4b76-ade3-cdb912c0b462/exporter/0.log" Jul 27 20:13:20.924867 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:20.924829 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qmvch_18301c03-eb72-4b76-ade3-cdb912c0b462/extractor/0.log" Jul 27 20:13:21.553833 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:21.553801 2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5b4zt/perf-node-gather-daemonset-kxjcs" Jul 27 20:13:23.193813 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.193782 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-kb4qg_248dea2e-c215-43ba-b4a7-bde0c0581a2d/manager/0.log" Jul 27 20:13:23.467502 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.467462 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bj7cf_f63021b5-abd9-47ce-95f8-540a45408c39/manager/0.log" Jul 27 20:13:23.675219 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.675183 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6f96b9564b-njdhs_0749ba4f-015e-4d0f-bc0d-dd26067f3140/manager/0.log" Jul 27 20:13:23.697602 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.697576 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-r7wzv_fa666236-ec46-48f4-af3c-32ce4e76e385/server/0.log" Jul 27 20:13:23.794480 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.794432 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-hkcrl_adacd43a-7371-4790-ae78-409decc5623b/manager/0.log" Jul 27 20:13:23.883504 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.883468 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-qml79_dc902e20-17ff-4d29-80fa-5acf84729993/manager/0.log" Jul 27 20:13:23.908156 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:23.908129 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d79566f86-qml79_dc902e20-17ff-4d29-80fa-5acf84729993/copy-manifests/0.log" Jul 27 20:13:24.632567 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.632531 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q_4d3437a2-8940-4738-8105-dd572305c90e/istio-proxy/0.log" Jul 27 20:13:24.810361 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.810337 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nqzgw_ddd49638-fe81-4f70-aa89-3858efdfacb2/kube-auth-proxy/0.log" Jul 27 20:13:24.830409 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.830366 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/0.log" Jul 27 20:13:24.865139 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.865107 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/1.log" Jul 27 20:13:24.895196 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.895141 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-djt9q_aeb8f814-39a2-4e03-9b40-5c6ab3981874/payload-pre-processing/0.log" Jul 27 20:13:24.918498 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.918468 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-n54d2_21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e/payload-processing/0.log" Jul 27 20:13:24.932448 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:24.932420 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85766f5879-5lm2d_6355fff2-3591-45c4-a7e3-d30721cf1b69/router/0.log" Jul 27 20:13:26.424867 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.424816 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-68cf4d799cl7z6q_4d3437a2-8940-4738-8105-dd572305c90e/istio-proxy/0.log" Jul 27 20:13:26.604588 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.604552 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-949fd4987-nqzgw_ddd49638-fe81-4f70-aa89-3858efdfacb2/kube-auth-proxy/0.log" Jul 27 20:13:26.628062 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.628005 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/0.log" Jul 27 20:13:26.667050 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.667012 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5ccb968c66-tqprs_61b8a2c6-049e-4451-9c7d-7a0670d98ba1/istio-proxy/1.log" Jul 27 20:13:26.703200 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.703082 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-djt9q_aeb8f814-39a2-4e03-9b40-5c6ab3981874/payload-pre-processing/0.log" Jul 27 20:13:26.744178 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.744137 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-n54d2_21d4d1a8-cef0-43d6-ac2b-7be3a3f7694e/payload-processing/0.log" Jul 27 20:13:26.773864 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:26.773812 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-85766f5879-5lm2d_6355fff2-3591-45c4-a7e3-d30721cf1b69/router/0.log" Jul 27 20:13:31.154305 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:31.154261 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-652sv_cbb1d9dd-f50e-496f-baaa-51ea94028342/kube-storage-version-migrator-operator/1.log" Jul 27 20:13:31.155141 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:31.155121 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-652sv_cbb1d9dd-f50e-496f-baaa-51ea94028342/kube-storage-version-migrator-operator/0.log" Jul 27 20:13:32.269302 ip-10-0-132-244 kubenswrapper[2566]: I0727 20:13:32.269261 2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-8ff5w_a6cd3b48-be1c-4d7c-aadb-4f6839cd75b0/kube-multus/0.log"